Evaluating features in an image possibly corresponding to an intersection of a pallet stringer and a pallet board

ABSTRACT

A programmable computer-implemented method is provided for finding possible corners of a pallet in an image. The method may comprise: acquiring a grey scale image including one or more pallets; determining, using a computer, horizontal cross correlations between the image and a first step-edge template to generate a set of horizontal cross correlation results; determining, using the computer, vertical cross correlations between the image and a second step-edge template to generate a set of vertical cross correlation results; and determining, using the computer, a first set of pixels, each such pixel respectively corresponding to a possible first corner of the one or more pallets, using a first corner template, the set of horizontal cross correlation results and the set of vertical cross correlation results.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication Ser. No. 61/548,776, filed Oct. 19, 2011, entitled “SYSTEMFOR IMAGING AND LOCATING A PALLET TO BE PICKED”; claims the benefit ofU.S. Provisional Patent Application Ser. No. 61/569,596, filed Dec. 12,2011, entitled “SYSTEM FOR IMAGING AND LOCATING A PALLET TO BE PICKED”;and claims the benefit of U.S. Provisional Patent Application Ser. No.61/709,611, filed Oct. 4, 2012, entitled “SYSTEM FOR IMAGING ANDLOCATING A PALLET TO BE PICKED,” the entire disclosure of each of theseapplications is hereby incorporated by reference herein.

FIELD OF THE INVENTION

The present invention relates generally to materials handling vehicles,and more particularly to a system for imaging and locating a pallet tobe picked by the vehicle.

BACKGROUND OF THE INVENTION

In a typical warehouse or distribution center, palletized stock itemsare stored in racks or other storage structures that are aligned to eachside of generally long, parallel extending aisles. To maximize availablespace, it is common for several storage structures to be verticallystacked, such that stock may be stored at elevated heights. Accordingly,an operator of a materials handling vehicle that is retrieving and/orputting away stock may be required to look upward from an operatingposition of the vehicle to identify the proper height and lateralposition of the forks for stock to be retrieved or put away.

Positioning a forklift carriage to pick up or to put away palletizedmaterials becomes increasingly more difficult at increasing heights.Visual perspective becomes more difficult. Extensive training can berequired in order to effectively perform the positioning adjustmentsnecessary. Even with sufficient ability, correct positioning can takemore time than desired for efficient use of the materials handlingvehicle and operator.

SUMMARY OF THE INVENTION

In accordance with a first aspect of the present invention, aprogrammable computer-implemented method is provided for findingpossible corners of a pallet in an image. The method may comprise:acquiring a grey scale image including one or more pallets; determining,using a computer, horizontal cross correlations between the image and afirst step-edge template to generate a set of horizontal crosscorrelation results; determining, using the computer, vertical crosscorrelations between the image and a second step-edge template togenerate a set of vertical cross correlation results; and determining,using the computer, a first set of pixels, each such pixel respectivelycorresponding to a possible first corner of the one or more pallets,using a first corner template, the set of horizontal cross correlationresults and the set of vertical cross correlation results.

The method may further comprise: comparing, using the computer, eachpixel in the first set of pixels to a first threshold value andgenerating a second set of pixels corresponding to a second set ofpossible first corners based on the comparison between the firstthreshold value and each pixel in the first set; and sliding, using thecomputer, a local maxima-locating window over the second set of pixelsand eliminating all pixels from the second set of pixels in eachplacement of the window except for the pixel having a maximum value suchthat a third set of pixels corresponding to a third set of possiblefirst corners are defined.

Wherein generating the second set of pixels may further comprisecomparing each pixel in the first set of pixels to the first thresholdvalue and eliminating pixels that are less than the first thresholdvalue so as to generate the second set of pixels.

The first corner template may be defined by an M×N window and thedetermining the first set of pixels may comprise: generating a firstcorner image having the first set of pixels by sliding the first cornertemplate window concurrently over the set of horizontal crosscorrelation results and the set of vertical cross correlation resultswherein; during each placement of the first corner template window as itslides concurrently over the horizontal and vertical cross correlationresults, receiving along one predefined vertical column of the windowone or more corresponding horizontal cross correlation results,receiving along one predefined horizontal row of the window one or morecorresponding vertical cross correlation results, identifying theminimum value of the cross correlation results in the one predefinedvertical column and the one predefined horizontal row, and assigning theidentified minimum value as a respective value in the first set ofpixels.

The one predefined vertical column may comprise one outside verticalcolumn and the one predefined horizontal row may comprise one outsidehorizontal row.

Assigning the identified minimum value may comprise assigning theidentified minimum value to the pixel of the first corner image which islocated at a position corresponding to an intersection of the onepredefined vertical column and the one predefined horizontal row.

The method may further comprise: determining, using the computer, afourth set of pixels, each such pixel respectively corresponding to apossible second corner of the one or more pallets using a second cornertemplate, the set of horizontal cross correlation results and the set ofvertical cross correlation results; comparing, using the computer, eachpixel in the fourth set of pixels to a second threshold value andgenerating a fifth set of pixels corresponding to a second set ofpossible second corners based on the comparison of the second thresholdvalue and each pixel in the fourth set; and sliding, using the computer,a local maxima-locating window over the fifth set of pixels andeliminating all pixels from the fifth set of pixels in each placement ofthe window except for the pixel having a maximum value such that a sixthset of pixels corresponding to a third set of possible second cornersare defined.

Wherein generating the fifth set of pixels may further comprisecomparing each pixel in the fourth set to the second threshold value andeliminating pixels that are less than the second threshold value so asto generate the fifth set of pixels.

The second corner template may be defined by an S×T window and thedetermining a fourth set of pixels may comprise: generating a secondcorner image having the fourth set of pixels by sliding the secondcorner template window concurrently over the set of horizontal crosscorrelation results and the set of vertical cross correlation resultswherein; during each placement of the second corner template window asit slides concurrently over the horizontal and vertical crosscorrelation results, receiving along one predefined inner verticalcolumn of the window one or more corresponding horizontal crosscorrelation results, negating those received horizontal crosscorrelation results, receiving along one predefined outside horizontalrow of the window one or more corresponding vertical cross correlationresults, identifying the minimum value of the cross correlation resultsin the one predefined vertical column and the one predefined horizontalrow, and assigning the identified minimum value as a respective value inthe fourth set of pixels.

In accordance with a second aspect of the present invention, a system isprovided for finding possible corners of a pallet in an image. Thesystem may comprise: an image acquisition component configured toacquire a grey scale image including one or more pallets; a computerconfigured to determine horizontal cross correlations between the imageand a first step-edge template to generate a set of horizontal crosscorrelation results; the computer further configured to determinevertical cross correlations between the image and a second step-edgetemplate to generate a set of vertical cross correlation results; andthe computer further configured to determine a first set of pixels, eachsuch pixel respectively corresponding to a possible first corner of theone or more pallets using a first corner template, the set of horizontalcross correlation results and the set of vertical cross correlationresults.

The computer may be further configured to compare each pixel in thefirst set of pixels to a first threshold value and to generate a secondset of pixels corresponding to a second set of possible first cornersbased on the comparison between the first threshold value and each pixelin the first set; and the computer may be further configured to slide alocal maxima-locating window over the second set of pixels and toeliminate all pixels from the second set of pixels in each placement ofthe window except for the pixel having a maximum value such that a thirdset of pixels corresponding to a third set of possible first corners aredefined.

Wherein to generate the second set of pixels, the computer may befurther configured to compare each pixel in the first set of pixels tothe first threshold value and eliminate pixels that are less than thefirst threshold value so as to generate the second set of pixels.

The first corner template may be defined by an M×N window and todetermine the first set of pixels the computer may be further configuredto: generate a first corner image having the first set of pixels bysliding the first corner template window concurrently over the set ofhorizontal cross correlation results and the set of vertical crosscorrelation results wherein; during each placement of the first cornertemplate window as it slides concurrently over the horizontal andvertical cross correlation results, the computer may be furtherconfigured to: receive along one predefined vertical column of thewindow one or more corresponding horizontal cross correlation results,receive along one predefined horizontal row of the window one or morecorresponding vertical cross correlation results, identify the minimumvalue of the cross correlation results in the one predefined verticalcolumn and the one predefined horizontal row; and assign the identifiedminimum value as a respective value in the first set of pixels.

The one predefined vertical column may comprise one outside verticalcolumn and the one predefined horizontal row may comprise one outsidehorizontal row.

The computer may be configured to assign the identified minimum value tothe pixel of the first corner image which is located at a positioncorresponding to an intersection of the one vertical column and the onehorizontal row.

The computer may be further configured to: determine a fourth set ofpixels, each such pixel respectively corresponding to a possible secondcorner of the one or more pallets, using a second corner template, theset of horizontal cross correlation results and the set of verticalcross correlation results; compare each pixel in the fourth set ofpixels to a second threshold value and generate a fifth set of pixelscorresponding to a second set of possible second corners based on thecomparison of the second threshold value and each pixel in the fourthset; and slide a local maxima-locating window over the fifth set ofpixels and eliminate all pixels from the fifth set of pixels in eachplacement of the window except for the pixel having a maximum value suchthat a sixth set of pixels corresponding to a third set of possiblesecond corners are defined.

Wherein to generate the fifth set of pixels, the computer may be furtherconfigured to compare each pixel in the fourth set to the secondthreshold value and eliminate pixels that are less than the secondthreshold value so as to generate the fifth set of pixels.

Wherein the second corner template is defined by an S×T window and todetermine the first set of pixels the computer may be further configuredto: generate a second corner image having the fourth set of pixels bysliding the second corner template window concurrently over the set ofhorizontal cross correlation results and the set of vertical crosscorrelation results, wherein during each placement of the second cornertemplate window as it slides concurrently over the horizontal andvertical cross correlation results, the computer may be configured to:receive along one predefined inner vertical column of the window one ormore corresponding horizontal cross correlation results, negate thosereceived horizontal cross correlation results, receive along onepredefined outside horizontal row of the window one or morecorresponding vertical cross correlation results, identify the minimumvalue of the cross correlation results in the one predefined verticalcolumn and the one predefined horizontal row; and assigning theidentified minimum value as a respective value in the fourth set ofpixels.

In accordance with a third aspect of the present invention, a computerprogram product for finding possible corners of a pallet in an image isprovided. The computer program product may comprise: a computer readablestorage medium having computer readable program code embodied therewith,the computer readable program code comprising: computer readable programcode configured to acquire a grey scale image including one or morepallets; computer readable program code configured to determinehorizontal cross correlations between the image and a first step-edgetemplate to generate a set of horizontal cross correlation results;computer readable program code configured to determine vertical crosscorrelations between the image and a second step-edge template togenerate a set of vertical cross correlation results; and computerreadable program code configured to determine a first set of pixels,each such pixel respectively corresponding to a possible first corner ofthe one or more pallets, using a first corner template, the set ofhorizontal cross correlation results and the set of vertical crosscorrelation results.

The computer program product may further comprise: computer readableprogram code configured to compare each pixel in the first set of pixelsto a first threshold value and generating a second set of pixelscorresponding to a second set of possible first corners based on thecomparison between the first threshold value and each pixel in the firstset; and computer readable program code configured to slide a localmaxima-locating window over the second set of pixels and eliminating allpixels from the second set of pixels in each placement of the windowexcept for the pixel having a maximum value such that a third set ofpixels corresponding to a third set of possible first corners aredefined.

The computer program product may further comprise: computer readableprogram code configured to compare each pixel in the first set of pixelsto the first threshold value and eliminate pixels that are less than thefirst threshold value so as to generate the second set of pixels.

Wherein the first corner template may be defined by an M×N window, andfurther comprising: computer readable program code configured togenerate a first corner image having the first set of pixels by slidingthe first corner template window concurrently over the set of horizontalcross correlation results and the set of vertical cross correlationresults; and computer readable program code configured to, during eachplacement of the first corner template window as it slides concurrentlyover the horizontal and vertical cross correlation results, receivealong one predefined vertical column of the window one or morecorresponding horizontal cross correlation results, receive along onepredefined horizontal row of the window one or more correspondingvertical cross correlation results, identify the minimum value of thecross correlation results in the one predefined vertical column and theone predefined horizontal row, and assign the identified minimum valueas a respective value in the first set of pixels.

The one predefined vertical column may comprise one outside verticalcolumn and the one predefined horizontal row may comprise one outsidehorizontal row.

The computer program product may further comprise: computer readableprogram code configured to assign the identified minimum value to thepixel of the first corner image which is located at a positioncorresponding to an intersection of the one predefined vertical columnand the one predefined horizontal row.

The computer program product may further comprise: computer readableprogram code configured to determine a fourth set of pixels, each suchpixel respectively corresponding to a possible second corner of the oneor more pallets using a second corner template, the set of horizontalcross correlation results and the set of vertical cross correlationresults; computer readable program code configured to compare each pixelin the fourth set of pixels to a second threshold value and generating afifth set of pixels corresponding to a second set of possible secondcorners based on the comparison of the second threshold value and eachpixel in the fourth set; and computer readable program code configuredto slide a local maxima-locating window over the fifth set of pixels andeliminate all pixels from the fifth set of pixels in each placement ofthe window except for the pixel having a maximum value such that a sixthset of pixels corresponding to a third set of possible second cornersare defined.

The computer program product may further comprise: computer readableprogram code configured to compare each pixel in the fourth set to thesecond threshold value and eliminate pixels that are less than thesecond threshold value so as to generate the fifth set of pixels.

The second corner template may be defined by an S×T window, and furthercomprising: computer readable program code configured to generate asecond corner image having the fourth set of pixels by sliding thesecond corner template window concurrently over the set of horizontalcross correlation results and the set of vertical cross correlationresults; and computer readable program code configured to, during eachplacement of the second corner template window as it slides concurrentlyover the horizontal and vertical cross correlation results, receivealong one predefined inner vertical column of the window one or morecorresponding horizontal cross correlation results, negate thosereceived horizontal cross correlation results, receive along onepredefined outside horizontal row of the window one or morecorresponding vertical cross correlation results, identify the minimumvalue of the cross correlation results in the one predefined verticalcolumn and the one predefined horizontal row, and assign the identifiedminimum value as a respective value in the fourth set of pixels.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a side view of a materials handling vehicle in which animage-based detection system of the present invention is incorporated;

FIG. 2 is a perspective view of a fork carriage apparatus forming partof the vehicle illustrated in FIG. 1;

FIG. 3 is a front view of a pallet;

FIG. 4 is a flow chart illustrating steps implemented by an imageanalysis computer in accordance with the present invention;

FIG. 5 is a flow chart illustrating steps implemented by the imageanalysis computer in accordance with the present invention foridentifying possible pallet corners;

FIGS. 6, 7, 6B and 7A illustrate first, second, third and fourth cornertemplates;

FIG. 6A illustrates the first corner template located over horizontaland vertical cross correlation results;

FIGS. 6C-6E illustrate an example for locating maximum pixel values in alower left corner image;

FIG. 6F illustrates an example of producing a lower resolution imagefrom a higher resolution image;

FIG. 6G illustrates a portion of a mid-level image to be furtheranalyzed for a location of a potential lower left corner;

FIG. 8 is a flow chart illustrating steps implemented by the imageanalysis computer in accordance with the present invention fordetermining a first Ro image and an optional second Ro image;

FIG. 9 illustrates an example for calculating a pixel value in ahorizontal gradient image;

FIG. 10 illustrates an example for calculating a pixel value in avertical gradient image;

FIGS. 11A-11D illustrate portions of a left Ro image including at leastone window;

FIGS. 12A, 12B, 13A and 13B are flow charts illustrating stepsimplemented by the image analysis computer in accordance with oneembodiment of the present invention for tracing a horizontal and avertical line from a lower left corner in the left Ro image;

FIGS. 14A and 14B illustrate portions of a left Ro image including atleast one window;

FIGS. 14C and 14D are flow charts illustrating steps in accordance withanother embodiment of the present invention for tracing a horizontal anda vertical line from a lower left corner;

FIG. 14E is a flowchart having some steps that are alternatives tocorresponding steps in the flowchart of FIG. 14C;

FIG. 15 is a flow chart illustrating steps implemented by the imageanalysis computer in accordance with the present invention fordetermining which of a plurality of possible lines is most likely to bean actual line passing through a prequalified lower left corner;

FIG. 15A illustrates two possible lines passing through a prequalifiedlower left corner; and

FIG. 16A illustrates a normalized gray scale image illustrating a palletP having a bottom pallet board 200 and a center stringer 208;

FIG. 16B is a horizontal cross correlation image corresponding to thegray scale image in FIG. 16A;

FIG. 16C is a vertical cross correlation image corresponding to the grayscale image in FIG. 16A;

FIG. 16D is a horizontal gradient image corresponding to the gray scaleimage in FIG. 16A;

FIG. 16E is a vertical gradient image corresponding to the gray scaleimage in FIG. 16A;

FIG. 16F is a first Ro image corresponding to the gray scale image inFIG. 16A;

FIG. 16G is a second Ro image corresponding to the gray scale image inFIG. 16A;

FIG. 17 is a view of a portion of a pallet illustrating lower left andright corners;

FIG. 18 illustrates a geometrical framework for evaluating potentialholes and a center stringer of a pallet structure in accordance with theprinciples of the present invention;

FIG. 19A illustrates a geometrical framework for estimating a height ofa center stringer in accordance with the principles of the presentinvention;

FIG. 19B illustrates a geometrical framework for evaluating possibleleft and right vertical edges of a center stringer in accordance withthe principles of the present invention;

FIG. 20A is a flow chart illustrating steps implemented by the imageanalysis computer in accordance with the present invention for analyzinga possible left edge of a center stringer;

FIG. 20B illustrates an example of pixels in an image portion analyzedaccording to the flowchart of FIG. 20A;

FIG. 21A is a flow chart illustrating steps implemented by the imageanalysis computer in accordance with the present invention fordetermining a height of a plurality of rectangles that potentiallyrepresent various portions of a pallet;

FIG. 21B illustrates a geometrical framework that graphically explainsaspects of the operation of the flowchart of FIG. 21A;

FIG. 22A illustrates an example of first rectangle that possiblyrepresents a fork opening located to the left of a pallet centerstringer;

FIG. 22B illustrates an example of second rectangle that possiblyrepresents a fork opening located to the right of a pallet centerstringer;

FIG. 22C illustrates an example of a third rectangle that possiblyrepresents a pallet center stringer;

FIG. 23A is a flow chart illustrating steps implemented by the imageanalysis computer in accordance with the present invention forcalculating the pixels that comprise the first second and thirdrectangles;

FIG. 23B illustrates a geometrical framework that graphically explainsaspects of the operation of the flowchart of FIG. 23A;

FIG. 23C is a flow chart illustrating steps implemented by the imageanalysis computer in accordance with the present invention forcalculating a vertical projection of negative pixels of the first andsecond rectangles;

FIG. 23D is a flow chart illustrating steps implemented by the imageanalysis computer in accordance with the present invention forcalculating a vertical projection of at least some of the negativepixels of the third rectangle;

FIG. 23E depicts a flowchart for an exemplary algorithm for maintainingObjectlists for candidate objects in an image frame in accordance withthe principles of the present invention;

FIG. 24A depicts a flowchart for an exemplary algorithm to eliminate atleast some of the candidate objects from an Objectlist for an imageframe in accordance with the principles of the present invention;

FIG. 24B depicts a flowchart for an exemplary algorithm to eliminate oneor more candidate objects that are nearby one another, in accordancewith the principles of the present invention;

FIG. 24C depicts a flowchart for an exemplary algorithm to eliminate oneor more candidate objects that are vertically local to one another, inaccordance with the principles of the present invention;

FIG. 24D depicts an example image frame with candidate objects inaccordance with the principles of the present invention;

FIG. 25 is a flow chart for tracking candidate objects between differentimage frames in accordance with the principles of the present invention;

FIGS. 26A and 26B depict a prior image frame and a next image frame withcandidate objects in accordance with the principles of the presentinvention;

FIG. 27A illustrates a pallet in the physical world with a location thatmay be calculated relative to more than one frame of reference;

FIG. 27B illustrates a pinhole camera model for projecting a physicalworld location onto an image plane;

FIG. 27C illustrates a gray scale image having different pixel locationsin accordance with the principles of the present invention;

FIG. 27D is a flow chart illustrating steps implemented by the imageanalysis computer in accordance with the present invention foridentifying outlier data;

FIG. 27E illustrates relative rotation of a camera coordinate systemaxes with respect to a world coordinate system axes in accordance withthe principles of the present invention;

FIG. 27F illustrates a pallet location in the physical world that, inaccordance with the principles of the present invention, may becalculated relative to more than one frame of reference;

FIG. 28 is a flow chart illustrating steps implemented by the imageanalysis computer in accordance with the present invention forpredicting state variables using a Kalman filter;

FIGS. 29A-29F are flowcharts illustrating for determining cutoutlocations in accordance with the principles of the present invention;and

FIG. 30 illustrates an image projection configuration of multiplepallets useful for performing extrinsic camera calibration in accordancewith the principles of the present invention

DETAILED DESCRIPTION OF THE INVENTION

Reference is now made to FIG. 1, which illustrates a fork lift truck orvehicle 10 in which the present invention may be incorporated. Thevehicle 10 comprises a power unit or main body 20 including anoperator's compartment 22. The vehicle 10 further comprises a mastassembly 30 and a fork carriage apparatus 40, shown best in FIG. 2.While the present invention is described herein with reference to astand-up counterbalanced truck, it will be apparent to those skilled inthe art that the invention and variations of the invention can be moregenerally applied to a variety of other materials handling vehiclesincluding a rider reach fork lift truck including a monomast, asdescribed in U.S. Patent Application Publication No. 2010/0065377, theentire disclosure of which is incorporated herein by reference. Thedifferent types of vehicle 10 in which embodiments of the presentinvention may be incorporated include manual vehicles, semi-autonomousvehicles, and autonomous vehicles.

The mast assembly 30 includes first, second and third mast weldments 32,34 and 36, wherein the second weldment 34 is nested within the firstweldment 32 and the third weldment 36 is nested within the secondweldment 34. The first weldment 32 is fixedly coupled to the truck mainbody 20. The second or intermediate weldment 34 is capable of verticalmovement relative to the first weldment 32. The third or inner weldment36 is capable of vertical movement relative to the first and secondweldments 32 and 34.

First and second lift ram/cylinder assemblies 35 (only the firstassembly is illustrated in FIG. 1) are coupled to the first and secondweldments 32 and 34 for effecting movement of the second weldment 34relative to the first weldment 32. Chains 35A (only the chain of thefirst assembly is illustrated in FIG. 1) are fixed to cylinders of thefirst and second lift assemblies and the third weldment 36 and extendover pulleys 35B (only the pulley of the first assembly is illustratedin FIG. 1) coupled to a corresponding one of the rams such that movementof rams of the first and second lift assemblies effects movement of thethird weldment 36 relative to the first and second weldments 32 and 34.

The fork carriage apparatus 40 is coupled to the third stage weldment 36so as to move vertically relative to the third stage weldment 36. Thefork carriage apparatus 40 also moves vertically with the third stageweldment 36 relative to the first and second stage weldments 32 and 34.

In the illustrated embodiment, the fork carriage apparatus 40 comprisesa fork carriage mechanism 44 to which the first and second forks 42A and42B are mounted, see FIGS. 1 and 2. The fork carriage mechanism 44 ismounted to a reach mechanism 46 which, in turn, is mounted to a mastcarriage assembly 48. The mast carriage assembly 48 is movably coupledto the third weldment 36. The reach mechanism 46 comprises a pantographor scissors structure 46A, which effects movement of the fork carriagemechanism 44 and the first and second forks 42A and 42B toward and awayfrom the mast carriage assembly 48 and the third weldment 36.

The fork carriage mechanism 44 comprises a carriage support structure44A and a fork carriage frame 44B. The forks 42A and 42B are mounted tothe fork carriage frame 44B. The frame 44B is coupled to the carriagesupport structure 44A for lateral and pivotable movement relative to thesupport structure 44A. A side-shift piston/cylinder unit 44C is mountedto the carriage support structure 44A and the fork carriage frame 44B,so as to effect lateral movement of the fork carriage frame 44B relativeto the carriage support structure 44A.

A tilt piston/cylinder unit 44D, shown only in FIG. 1, is fixedlyattached to the carriage support structure 44A and contacts the forkcarriage frame 44B for effecting pivotable movement of the fork carriageframe 44B relative to the carriage support structure 44A.

An operator standing in the compartment 22 may control the direction oftravel of the truck 10 via a tiller 12. The operator may also controlthe travel speed of the truck 10, fork carriage apparatus and mastassembly extension, and tilt and side shift of the first and secondforks 42A and 42B via a multifunction controller 14.

In accordance with the present invention, an image-based palletdetection system 100 is provided for repeatedly capturing images as thefork carriage apparatus 40 is raised and lowered, identifying one ormore objects in image frames which may comprise one or more pallets,determining which of the one or more objects most likely comprises apallet, tracking the one or more objects in a plurality of frames so asto determine their locations relative to the fork carriage apparatus 40and generating and transmitting pallet location information to a vehiclecomputer 50 located on the vehicle power unit 20.

The system 100 comprises an image analysis computer 110 coupled to thefork carriage frame 44B, a light source 120 coupled to a lower section144B of the fork carriage frame 44B, an imaging camera 130 coupled tothe lower section 144B of the fork carriage frame 44B and a triggerswitch 140 located in the operator's compartment 22 to actuate thesystem 100. While some mounting positions may be more preferable thanothers, the imaging camera and lights may be located either above orbelow the forks. It is desirable that the camera side shift with theforks, but it does not necessarily need to tilt with the forks. Howeverthe camera and lights are mounted, the forks should be in the bottom ofthe field of view to give maximum warning of an approaching pallet. Inthe illustrated embodiment, the imaging camera, 130 and the lights, 120,are below the forks, see FIG. 1. The computer 110 may be locatedanywhere on the truck. It is also possible that items 110, 120, and 130may be combined into a single package, comprising a smart camera. Thelight source 120, in the illustrated embodiment, comprises a nearinfrared light source which can, for example, be one or more lightemitting diodes (LEDs) that are on while the forks 42A and 42B aremoving or are “pulsed” so as to be on only while an image is beingacquired. The image analysis computer 110 may either with a wired cableor wirelessly transmit pallet identification and location information tothe vehicle computer 50 such that the vehicle computer 50 may accuratelyposition the forks 42A in 42B in vertical and lateral directions, asdefined by a Y-axis and X-axis, respectfully, see FIG. 1.

The vehicle and image analysis computers 50 and 110 may comprise anykind of a device which receives input data, processes that data throughcomputer instructions, and generates output data. Such a computer can bea hand-held device, laptop or notebook computer, desktop computer,microcomputer, digital signal processor (DSP), mainframe, server, cellphone, personal digital assistant, other programmable computer devices,or any combination thereof. Such computers can also be implemented usingprogrammable logic devices such as field programmable gate arrays(FPGAs) or, alternatively, realized as application specific integratedcircuits (ASICs) or similar devices. The term “computer” is alsointended to encompass a combination of two or more of the above reciteddevices, e.g., two or more microcomputers. The vehicle and imageanalysis computers 50 and 110 may be connected wirelessly or hard-wiredto one another. It is also contemplated that the computers 50 and 110may be combined as a single computer. Accordingly, aspects of thepresent invention may be implemented entirely as hardware, entirely assoftware (including firmware, resident software, micro-code, etc.) or ina combined software and hardware implementation that may all generallybe referred to herein as a “circuit,” “module,” “component,” or“system.” Furthermore, aspects of the present invention may take theform of a computer program product embodied in one or more computerreadable media having computer readable program code embodied thereon.

When an operator wishes to pick a pallet P, the operator maneuvers thevehicle 10 such that it is positioned directly in front of and generallylaterally aligned in the X-direction with a desired pallet P on a rack Rto be picked, see FIG. 1. The operator then raises the forks 42A and 42Bvertically in the Y-direction via actuation of the multifunctioncontroller 14 to a position above the last pallet P which is to beignored. The image analysis computer 110 causes the imaging camera 130to take image frames, such as at a rate of 10-30 fps (frames/second), asthe fork carriage apparatus 40 continues to move vertically. As will bediscussed in greater detail below, the image analysis computer 110analyzes the images, identifies one or more objects in the image frames,determines which of the one or more objects most likely comprises apallet, tracks the one or more objects in a plurality of image frames,determines the location of the objects relative to a world coordinateorigin and generates and transmits pallet location information to thevehicle computer 50. The image analysis computer 110 may also wirelesslytransmit to the vehicle computer 50 pallet location information so thatthe vehicle computer 50 may accurately position the forks 42A and 42B invertical and lateral directions such that the forks 42A and 42B arepositioned directly in front of openings in the pallet. Thereafter, anoperator need only cause the fork carriage apparatus to move toward thepallet such that the forks enter the pallet openings.

In FIG. 3, a pallet P is illustrated having bottom and top pallet boards200 and 202 first and second outer stringers 204 and 206 and a centerstringer 208. The boards 200, 202 and stringers 204, 206 and 208 definefirst and second fork receiving openings 210 and 212. The pallet boards200 and 202 and the center stringer 208 also define a lower left corner214, a lower right corner 216, an upper left corner 218 and an upperright corner 220. The lower left corner 214 is also referred to hereinas the lower left corner of the center stringer 208 and the lower rightcorner 216 is also referred to herein as the lower right corner of thecenter stringer 208.

In order to identify one or more objects in each image frame taken bythe imaging camera 130 and determine which of the one or more objectsmost likely comprises a pallet, assuming that the camera is orientedlooking either up or straight ahead, either above or below the forks,the image analysis computer 110 implements at least the followingprocesses: identifies possible corners of one or more pallets P in agray scale image, see FIG. 4 process 230; determines first and anoptional second Ro images including orthogonal distance(s) from anorigin point to one or more possible lines on one or more possiblepallets in the corresponding gray scale image, see FIG. 4 process 232;attempts to trace horizontal and vertical lines from each possible lowerleft corner of a third set of possible lower left corners to define aset of prequalified lower left corners, see FIG. 4 process 234;determines, for each prequalified lower left corner, which of aplurality of possible lines is most likely to be an actual line passingthrough a prequalified lower left corner, see FIG. 4 process 236; andattempts to identify a set of lower right corners for each prequalifiedlower left corner for which an actual line is found, see FIG. 4 process238. It is further contemplated that an imaging camera may be locatedabove the forks 42A, 42B looking down instead of up, which occurs whenthe camera is located below the forks 42A, 42B. In such an embodiment,the computer 110 may attempt to trace horizontal and vertical lines fromeach possible upper left corner of a third set of possible upper leftcorners to define a set of prequalified upper left corners, anddetermine, for each prequalified upper left corner, which of a pluralityof possible lines is most likely to be an actual line passing through aprequalified upper left corner. The computer 110 may also attempt tolocate a third set of possible upper right corners associated with eachupper left corner.

The process 230 for identifying possible corners of one or more palletsP in an image will now be described with reference to FIG. 5. In theillustrated embodiment, the data defining each image frame received fromthe imaging camera 130 comprises gray scale data. The image analysiscomputer 110 normalizes the gray scale data to make it zero-mean data,see step 240 in FIG. 5. Creation of a normalized gray scale image can beaccomplished by calculating the mean value of all the pixels in the grayscale image and then subtracting that mean value from each individualpixel value. As a consequence, pixels in the normalized gray scale imagecorresponding to pallet structure tend to have positive pixel valueswhile other pixels in the normalized gray scale image tend to havenegative pixel values. The term “gray scale data” and “gray scale image”used henceforth means “normalized gray scale data” and “normalized grayscale image,” respectively. The image analysis computer 110 thendetermines a set of horizontal cross correlation results, i.e., ahorizontal cross correlation image, see step 242, using a firststep-edge template comprising in the illustrated embodiment:T(x)=−1,−1,−1,−1,−1,−1,+1,+1,+1,+1,+1,+1,

wherein “−1” corresponds to an open area and “+1” corresponds tostructure on the pallet.

Using the first step-edge template T(x) and the normalized gray scaleimage data, the image analysis computer 110 determines the horizontalcross correlation results from the following single dimension normalizedcross correlation equation:

${N\; C\;{C(x)}} = \frac{\sum\limits_{W{(x)}}{\left( {f_{x} - \overset{\_}{f_{x}}} \right){T(x)}}}{\sqrt{\sum\limits_{W{(x)}}{\left( {f_{x} - \overset{\_}{f_{x}}} \right)^{2}{\sum\limits_{W{(x)}}{T^{2}(x)}}}}}$

wherein:

W(x)=dimensional limits of the first step-edge template, i.e., 12 in theillustrated embodiment.

f_(x)=a gray scale intensity value at a single point in the gray scaleimage;

f_(x) =an average of the intensity values over W(x); and

T(x)=first step-edge template.

The horizontal cross correlation image has the same number of pixels asits corresponding gray scale image such that there is a one-to-onecorrespondence between pixel locations in the gray scale image and thehorizontal cross correlation image. The horizontal cross correlationimage provides information regarding where a region of the gray scaleimage is similar to the first step-edge template, with the firststep-edge template positioned horizontally over the image region. Withthe normalized gray scale image data having values from −m to +n, eachpixel in the horizontal cross correlation image has a value between −1and +1, wherein pixels having values of −1 correspond to a perfectmiss-match between the template and gray scale pixels over which thetarget is positioned and pixels having values of +1 correspond to aperfect match between the template and gray scale pixels over which thetemplate is positioned. Hence, positive horizontal cross correlationvalues near or at +1 correspond to a vertical interface between an openarea and an adjacent pallet structure, i.e., the open area is on theleft and the pallet structure is on the right, wherein the interfacebetween the open area and the pallet structure extends in the Y orvertical direction.

FIG. 16A illustrates a normalized gray scale image illustrating a palletP having a bottom pallet board 200 and a center stringer 208. FIG. 16Bis a horizontal cross correlation image corresponding to the gray scaleimage in FIG. 16A. In FIG. 16B, regions 1010 have positive pixel valuesand regions 1020 have negative pixel values. Regions 1012 have highmagnitude positive values near or at +1 such that they may correspond toa transition in the gray scale image between an open area and avertically oriented edge of a pallet. Hence, region 1012A corresponds toan interface between pallet opening 210 and a left surface 208A of thecenter stringer 208.

The image analysis computer 110 then determines a set of vertical crosscorrelation results, i.e., a vertical cross correlation image, see step244, using a second step-edge template comprising in the illustratedembodiment:T(y)=−1,−1,−1,−1,−1,−1,−1,−1,−1,−1,−1,−1,+1,+1,+1,+1,+1,+1,+1,+1,+1,+1,+1,+1,

wherein “−1” corresponds to an open area and “+1” corresponds tostructure on the pallet.

Using the second step-edge template T(y) and the image gray scale data,the image analysis computer 110 determines the vertical crosscorrelation results from the following single dimension normalized crosscorrelation equation:

${{NCC}(y)} = \frac{{\underset{W{(y)}}{\Sigma}\left( {f_{y} - \overset{\_}{f_{y}}} \right)}{T(y)}}{\sqrt{{\underset{W{(y)}}{\Sigma}\left( {f_{y} - \overset{\_}{f_{y}}} \right)}^{2}\underset{W{(x)}}{\Sigma}{T^{2}(y)}}}$

wherein:

W(y)=the dimensional limits of the template, i.e., 24 in the illustratedembodiment;

f_(y)=a gray scale intensity value at a single point in the gray scaleimage;

f_(y) =an average of the intensity values over W(y);

T(y)=second step-edge template.

The vertical cross correlation image has the same number of pixels asits corresponding gray scale image such that there is a one-to-onecorrespondence between pixel locations in the gray scale image and thevertical cross correlation image. The vertical cross correlation imageprovides information regarding where a region of the gray scale image issimilar to the second step-edge template, with the second step-edgetemplate positioned vertically over the image region such that thenegative template values are positioned vertically above the positivetemplate values. With the normalized gray scale image data having valuesfrom −m to +n, each pixel in the vertical cross correlation image has avalue between −1 and +1, wherein pixels having values at −1 correspondto a perfect miss-match between the template T(y) and the gray scalepixels over which the template is positioned and pixels having values at+1 correspond to a perfect match between the template T(y) and grayscale pixels over which the template is positioned. Hence, positivevertical cross correlation values at +1 identify a perfect horizontalinterface between an open area and an adjacent pallet structure, i.e.,the open area is above the pallet structure, wherein the horizontalinterface between the open area and the pallet structure extends in theX or horizontal direction and −1 denotes the exact opposite.

FIG. 16C is a vertical cross correlation image corresponding to the grayscale image in FIG. 16A. In FIG. 16C, regions 1030 have positive pixelvalues and regions 1040 have negative pixel values. Regions 1032 havehigh magnitude positive values such that they generally correspond to atransition in the gray scale image between an open area and ahorizontally oriented edge of a pallet. Hence, region 1032A correspondsto an interface between an open area and an upper surface 200A of bottompallet board 200.

As described above, the horizontal cross-correlation image and thevertical cross-correlation image may have the same number of pixels asthe normalized gray scale image. This means that a correspondinghorizontal and vertical cross-correlation value is calculated for everypixel in the gray scale image. Alternatively, the respectivecross-correlation images may be calculated in a way that each such imagemay have fewer numbers of pixels than the normalized gray scale image.

For example, the imaging camera 130 may be configured so that the grayscale image captures a view that extends, horizontally, from a firstpoint that is substantially in line with the outside edge of the secondor left fork 42B to a second point that is substantially in line withthe outside edge of the first or right fork 42A. Thus, there is an innerregion of the gray scale image that can be described as being “betweenthe forks” 42A and 42B. It is within this between the forks region thatcorners are sought.

An exemplary gray scale image may have 480 columns and 752 rows ofpixels. If the origin point for this gray scale image is defined asbeing the top left pixel location, then the left-most column of pixelshas an x coordinate value of “1” and the right-most column of pixels hasan x coordinate value of “480”. Similarly, in the vertical direction,the top-most row of pixels may have a y coordinate value of “1” and thebottom-most row of pixels may have a y coordinate value of “752”. Forthe gray scale image, a left limit for x coordinates may be chosen thatcorresponds to a point that is substantially in line with the inner edgeof the left fork 42B; and a right limit for x coordinates may be chosenthat corresponds to a point that is substantially in line with the inneredge of the right fork 42A. Any pixel in the gray scale image that hasits x coordinate between the left and right limits is considered to be“between the forks”. For example, the left limit may be “80” and theright limit may be “400”, but other values may alternatively be selectedwithout departing from the scope of the present invention.

Thus, in the description above with respect to calculating a horizontalcross correlation image and a vertical cross correlation image, thecalculations may be limited to only those pixels that are locatedbetween the forks such that the horizontal cross correlation image andthe vertical cross correlation image each have fewer columns of pixelsthan the gray scale image.

The image analysis computer 110 next determines pixel valuescorresponding to a first set of possible first corners of one or morepallets, i.e., a lower-left corner image, using a first corner template,the horizontal cross correlation results and the vertical crosscorrelation results, see step 246 in FIG. 5. In the illustratedembodiment, the possible first corners comprise possible lower leftcorners of one or more pallets. The first corner template 300 is definedby an M×N window, a 3×3 window in the illustrated embodiment,corresponding to a lower-left corner of a pallet, see FIG. 6.

The computer 110 slides the first corner template 300 concurrently overthe set of horizontal or first cross correlation results or image 290and the set of vertical or second cross correlation results or image 292so as to determine pixel values for the lower-left corner image, seeFIG. 6A. More specifically, the first corner template 300 is positionedover the horizontal and vertical cross correlations so that the requiredvalues from each can be retrieved. Thus, the pixel location 304 is movedover similarly positioned pixels in the first and second crosscorrelation images. As the template pixel location 304 is sequentiallymoved over each pixel in the first cross correlation image and asimilarly located pixel in the second cross correlation image, a pixelvalue, Score_(LowerLeftCorner), for that pixel location is defined forthe lower-left corner image. During each placement of the first cornertemplate 300, the computer 110 places into first and second pixels 301Aand 301B of an outer vertical column 301 corresponding horizontal crosscorrelation results or pixel values, i.e., pixel values havingcorresponding locations in the horizontal cross correlation image, andplaces into first and second pixels 302A and 302B of an outer horizontalrow 302 corresponding vertical cross correlation results or pixelvalues, i.e., pixel values having corresponding locations in thevertical cross correlation image, see FIG. 6. No values are placed intothe pixels 303. The computer 110 then determines the lowest or minimumvalue of pixels 301A, 301B, 302A and 302B, wherein the minimum is afuzzy logic AND, and assigns that lowest or minimum value to pixel 304at the intersection of the vertical and horizontal columns 301 and 302,see FIG. 6, which pixel 304 should have a value between −1 and +1.Values closer to “−1” and “+1” indicate that the pixel has a stronglikelihood of defining a transition between structure on a pallet andopen area or vice versa in the gray scale image. Values close to “0”indicate that the pixel has a strong likelihood of defining a continuousarea of the normalized gray scale image corresponding to continuouspallet structure or a continuous open area. Values close to “0” aretypically ignored as the image analysis computer 110 attempts to find,as noted below, pixels having values near +1. The value assigned topixel 304 is also assigned to a corresponding pixel location in thelower-left corner image, i.e., the value from pixel 304 is assigned to apixel location in the lower-left corner image corresponding to the pixellocation in each of the first and second cross correlation images overwhich the template pixel location 304 is positioned. The pixels in theresulting lower left corner image define a first set of pixelscorresponding to a first set of possible first or lower left corners. Inan embodiment, the number of first set pixels in the lower left cornerimage equals the number of pixels in each of the gray scale images, thehorizontal cross correlation image and the vertical cross correlationimage. Alternatively, similar to the technique described above withrespect to the cross correlation images, the determination of the pixelvalues in the lower left corner image may be limited to only those pixellocations that are between the forks (i.e., between the left limit andthe right limit, as described above). In this instance, the lower leftcorner image, the vertical cross correlation image and the horizontalcross correlation image would all have the same number of pixels butthis number would be less than the number of pixels in the gray scaleimage.

For one or more of the vertical column first and second pixels 301A and301B and the horizontal column first and second pixels 302A and 302Bwhich fall outside of the boundary of the horizontal and vertical crosscorrelation results or images 290 and 292, the computer 110 “pads out”the horizontal and vertical cross correlations by placing a 0 in each ofthose pixels. Hence, in the example illustrated in FIG. 6A, a 0 isplaced into pixels 301A and 302A.

In order to reduce the number of first corner pixels, the image analysiscomputer 110 compares the value of each pixel in the first set of pixelsto a first threshold value, such as 0.5 in the illustrated embodiment,see step 248 in FIG. 5. All pixels in the first set having a value lessthan the first threshold value are eliminated, i.e., the value for eacheliminated pixel is changed to 0. All remaining pixels, i.e., thosehaving values equal to or greater than the first threshold value, definea second set of pixels corresponding to a second set of possible firstcorners. Pixels having a value less than 0.5 are deemed not to bepossible first corners as pixels near +1 indicate either a verticalinterface between an open area on the left and pallet structure on theright or a horizontal interface between an upper open area and a lowerpallet structure.

In order to further reduce the number of first corner pixels, the imageanalysis computer 110 recursively slides a local maxima-locating window,e.g., an M×M window, where M is equal to or greater than 3, over thesecond set of pixels and eliminates all pixels from the second set ofpixels in each placement of the window except for the pixel having amaximum value. More specifically, the behavior of the maxima-locatingwindow is as if it is sequentially moved such that a center pixel of themaxima-locating window is positioned over each pixel in the second setof pixels. During each positioning of the maxima-locating window, allpixels are eliminated (i.e., set to “0”) except for the pixel having amaximum value. After the window has been sequentially moved over allpixels in the second set of pixels, the remaining pixels define a thirdset of pixels corresponding to a third set of possible first corners,see step 250 in FIG. 5. However, in practice, an identical result can beachieved but with far less computational complexity.

FIGS. 6C-6E illustrate a sequence of an efficient method for moving a5×5 window, for example, to identify the pixels in the second set ofpixels having a maximum value. In FIG. 6C, merely a portion of a largeimage comprising the second set of pixels is depicted. In particular,the image portion 702 of FIG. 6C is 11 columns×6 rows of pixels whereineach pixel has a value that was determined as described earlier using a3×3 corner template and, further, with some having a value greater thanthe first threshold value, 0.5. FIGS. 6C-6E include a series of“snapshots” of the image portion 702 to illustrate how the 5×5 window isiteratively moved in order to efficiently locate the maximum pixels.Each snapshot of the image portion 702 corresponds to an iterative stepof moving the 5×5 window 704; the step number is located in the leftcolumn 706.

Initially, as depicted in the first snapshot, the 5×5 window 704 isplaced in the upper left corner of the image portion 702. The upper leftcorner of the image portion 702 may have its x coordinate position at alocation corresponding to the left most column of the gray scale image.However, if processing of the lower left corner image by the imageanalysis computer 110 is limited to those pixels that are “between theforks,” then the x coordinate position of the upper left corner of theimage portion 702 is determined by the value of the left limit for xcoordinates (e.g., 80). The 25 pixels are evaluated to determine thatthe pixel at location (2, 2), having a value of 0.95, is the maximumvalued pixel. The values of all the other pixels in the 5×5 window 704are set to zero. The location and value of the maximum pixel areretained for later use.

The 5×5 window 704 is then moved one pixel to the right as shown in thesecond snapshot. The four left-most columns 708 of the 5×5 window 704have pixels that, other than the maximum value, if present, are allzero. Thus, in “Step 2”, the 5 values in the right-most column 710 arecompared to determine if any are greater than the previously identifiedmaximum value, presuming the previously identified maximum value iswithin the bounds of the 5×5 window 704. In this second snapshot, thepreviously identified maximum value is within the bounds of the window704. Hence, the 5 values in the right-most column 710 are compared to0.95, i.e., the previously identified maximum value. It is not necessaryto consider the values of the remaining 19 pixels in the four left-mostcolumns 708. In this instance, none of the pixels in the right-mostcolumn 710 are greater than 0.95. Because the location of the maximumvalue (e.g., pixel (2,2)) lies within the span of the 5×5 window 704,all the pixels in column 710 are set to zero and the maximum pixel valueand location remain the same.

In the third snapshot, the 5×5 window 704 has once again been moved onecolumn to the right. The pixels in the current placement of theright-most column 710 are not evaluated against the previouslyidentified maximum value because it is no longer within the 5×5 window704. In this snapshot, the pixels within the right-most columns 710 areonly compared to one another. Under these conditions, the pixel atlocation (7, 1) becomes the new maximum value pixel having a value of0.8.

The fourth snapshot occurs when the 5×5 window 704 is again moved onepixel to the right. In this instance none of the pixels in the currentplacement of the right-most column are greater than the current maximumvalue. Because the current maximum value pixel is also within theboundary of the 5×5 window 704, the maximum value pixel location andpixel value remain unchanged. Further, all pixels in the right-mostcolumn are set to zero.

The fifth and sixth steps occur to produce the 5×5 window 704 placementshown in the fifth and sixth snapshots of FIG. 6D. In the sixth step,the right-most column aligns with column 10 of image portion 702. Inthis column, there is a pixel at location (10,5) that is greater thanthe current maximum value “0.8” and the current maximum value is stilllocated (e.g., pixel (7,1)) within the boundary of the 5×5 window 604.Under these conditions the following changes occur: the previous maximumvalue pixel location (7,1) is set to zero, the current maximum pixelvalue is identified as “1”, and the current maximum pixel value locationis identified as (10,5).

After the next iteration, the 5×5 window 704 is located at the rightmost portion of the image portion 702. Eventually, the 5×5 window 704will reach a point where the right-most column 710 is aligned with theright-most column of the full image. Once this occurs, the nextiteration of moving the window 704 produces the window placement shownin the snapshot at the bottom of FIG. 6D. In other words, when theright-most edge of the image is reached, the 5×5 window 704 is returnedto the left-most edge of the image but shifted down by one row. Ifprocessing of the lower left corner image by the image analysis computer110 is limited to those pixels that are “between the forks,” then the xcoordinate position of the right most column of the image portion 702 isdetermined by the value of the right limit for x coordinates (e.g.,400).

When the 5×5 window 704 is placed, as shown in the bottom snapshot ofFIG. 6D, so that its left-most column aligns with the left-most columnof the image (which may be determined by the left limit for xcoordinates (e.g., 80)), the pixels of interest are not the right-mostcolumn of the window 704 but are the bottom-most row 712. Recalling theinitially saved maximum value “0.95” at location (2,2), the pixels inthe bottom-most row 712 are compared to see if any are greater than thatmaximum value, presuming that previous maximum value pixel is locatedwithin the current boundary of the 5×5 window 704. In this instance,there are no pixels greater than “0.95” which causes all the pixels inthe bottom-most row 712 to be set to zero. Thus, the maximum valueremains at “0.95” at location (2,2) and this information is again savedso that it can be used when the 5×5 window 704 once again returns to theleft-most column of the image.

In the next iteration, the 5×5 window 704 is moved one pixel to theright and the focus returns to the pixels in the right-most column 710.These pixels are compared with the current maximum value and pixelslocation to determine what pixels may need to be modified. The movementof the window 704 continues as before until the 5×5 window reaches thebottom-most and right-most pixel of the image. At that point, a matrixof pixels defining a third set of possible first corners exists havingthe same dimensions as the lower left corner image and each pixel with anon-zero value has a value that represents a confidence score that thepixel corresponds to a possible lower left corner of a pallet structurein the normalized gray scale image.

The image analysis computer 110 next determines pixel valuescorresponding to a first set of possible second corners of the one ormore pallets, i.e., a lower-right corner image, using a second cornertemplate, the horizontal cross correlation results and the verticalcross correlation results, see step 252 in FIG. 5. In the illustratedembodiment, the possible second corners comprise possible lower rightcorners. The second corner template 310 is defined by an S×T window, a3×3 window in the illustrated embodiment, corresponding to a lower-rightcorner of a pallet, see FIG. 7.

The computer 110 slides the second corner template 310 concurrently overthe set of horizontal cross correlation results and the set of verticalcross correlation results so as to define pixel values,Score_(LowerRightCorner), for the lower-right corner image. Morespecifically, the second corner template 310 is positioned over thehorizontal and vertical cross correlations so that required values fromeach can be retrieved. Thus, the pixel location 312C is movedsequentially over similarly positioned pixels in the first and secondcross correlation images, see FIG. 7. As the template pixel location312C is sequentially moved over each pixel in the first crosscorrelation image and a similarly located pixel in the second crosscorrelation image, a value for that pixel location is defined for thelower-right corner image. During each placement of the second cornertemplate window 310, the computer 110 places into first and secondpixels 311A and 311B of a center vertical column 311 correspondinghorizontal cross correlation results or pixel values, i.e., pixel valueshaving corresponding locations in the horizontal cross correlationimage, and places into first and second pixels 312A and 312B of an outerhorizontal row 312 corresponding vertical cross correlation results orpixels, i.e., pixel values having corresponding locations in thevertical cross correlation image. The computer 110 also negates thevalues added to the first and second pixels 311A and 311B. Values closeto −1 in the horizontal cross correlation image correspond to aninterface between a pallet structure on the left and an open area on theright. By negating the values in the first and second pixels 311A and311B, high magnitude negative values in the horizontal cross correlationimage are converted to high magnitude positive values. Hence, verticaltransitions between a pallet structure on the left and an open area onthe right are defined by high magnitude positive values, which isbeneficial when finding fifth and sixth sets of pixels in the mannerdiscussed below. No values are placed into pixels 313. The computer 110then determines the lowest or minimum value of pixels 311A, 311B, 312Aand 312B, wherein the minimum is a fuzzy logic AND, and assigns thatlowest or minimum absolute value to the pixel 312C of the horizontal row312, see FIG. 7. The resulting value in pixel 312C should have a valuebetween −1 and +1. Values closer to “−1” and “+1” indicate that thepixel has a strong likelihood of defining a transition between an openarea and structure on a pallet or vice versa in the gray scale image.Values close to “0” indicate that the pixel has a strong likelihood ofdefining a continuous area of the normalized gray scale imagecorresponding to continuous pallet structure or a continuous open area.In an embodiment, the value assigned to pixel 312C is also assigned to acorresponding pixel location in the lower right corner image.Alternatively, similar to the technique described above with respect tothe lower left corner image, the determination of the pixel values inthe lower right corner image may be limited to only those pixellocations that are between the forks (i.e., between the left limit andthe right limit, as described above). In this instance, the lower rightcorner image, the vertical cross correlation image and the horizontalcross correlation image would all have the same number of pixels butthis number would be less than the number of pixels in the gray scaleimage. The pixels in the resulting lower right corner image define afourth set of pixels corresponding to a first set of possible second orlower right corners. The number of fourth set pixels in the lower rightcorner image equals the number of pixels in each of the gray scaleimage, the horizontal cross correlation image and the vertical crosscorrelation image.

The image analysis computer 110 compares the value of each pixel in thefourth set of pixels to a second threshold value, such as 0.5, see step254 in FIG. 5. All possible second corner pixels having a value lessthan the second threshold value are eliminated, i.e., the value for eacheliminated second corner pixel is changed to 0. All remaining secondcorner pixels, i.e., those having values equal to or greater than thesecond threshold value, define a fifth set of pixels corresponding to asecond set of possible second corners. Pixels having a value less than0.5 are deemed not to be possible first corners as pixels near +1indicate either a vertical interface between a pallet structure on theleft and an open area on the right or a horizontal interface between anupper open area and a lower pallet structure.

The image analysis computer 110 recursively slides a localmaxima-locating window, e.g., an M×M window, where M is equal to orgreater than 3, over the fifth set of pixels and eliminates all pixelsin each placement of the window except for the pixel having a maximumvalue such that a sixth set of pixels is defined corresponding to athird set of possible second corners, see step 256 in FIG. 5. Morespecifically, the maxima-locating window is sequentially moved such thata center pixel of the maxima-locating window is positioned over eachpixel of the fifth set of pixels. During each positioning of themaxima-locating window, all pixels in the fifth set are eliminatedexcept for the pixel having a maximum value. The remaining pixels definea sixth set of pixels corresponding to a third set of possible secondcorners. The method discussed above with regards to FIGS. 6C-6E todetermine the third set of pixels corresponding to a third set ofpossible first corners may be used to define a sixth set of pixelscorresponding to a third set of second corners.

As noted above, in a further contemplated embodiment, an imaging cameramay be located above the forks 42A, 42B looking down instead of lookingup or straight ahead. In this embodiment, the computer 110 may attemptto locate a plurality of possible upper left corners and a plurality ofpossible upper right corners instead of locating possible lower left andlower right corners.

The image analysis computer 110 determines pixel values corresponding toa first set of possible upper left corners of the one or more palletsusing a third corner template 500, the horizontal cross correlationresults and the vertical cross correlation results, see FIG. 6B. Thethird corner template 500 is defined by a 3×3 window in the illustratedembodiment, corresponding to an upper left corner of a pallet.

The computer 110 slides the third corner template 500 concurrently overthe set of horizontal cross correlation results and the set of verticalcross correlation results so as to define pixel values for an upper leftcorner image. More specifically, the third corner template 500 ispositioned over the horizontal and vertical cross correlations so thatrequired values from each can be retrieved. Thus, a pixel location 501is moved sequentially over similarly positioned pixels in the first andsecond cross correlation images, see FIG. 6B. As the template pixellocation 501 is sequentially moved over each pixel in the first crosscorrelation image and a similarly located pixel in the second crosscorrelation image, a value for that pixel location is defined for theupper left corner image. During each placement of the third cornertemplate window 500, the computer 110 places into first and secondpixels 503A and 503B of an outer vertical column 504 correspondinghorizontal cross correlation results or pixel values, i.e., pixel valueshaving corresponding locations in the horizontal cross correlationimage, and places into first and second pixels 505A and 505B of a middlehorizontal row 506 corresponding vertical cross correlation results orpixels, i.e., pixel values having corresponding locations in thevertical cross correlation image. The computer 110 also negates thevalues added to the first and second pixels 505A and 505B. The computer110 then determines the lowest or minimum value of pixels 503A, 503B,505A and 505B and assigns that lowest or minimum absolute value to thepixel 501 of the vertical row 504, see FIG. 6B. The resulting value inpixel 501 should have a value between −1 and +1. The pixels in theresulting upper left corner image define a first set of pixelscorresponding to a first set of possible upper left corners.

The image analysis computer 110 compares the value of each pixel in thefirst set of upper left corner pixels to a third threshold value, suchas 0.5. All possible upper left corner pixels having a value less thanthe third threshold value are eliminated, i.e., the value for eacheliminated upper left corner pixel is changed to 0. All remaining upperleft corner pixels, i.e., those having values equal to or greater thanthe third threshold value, define a second set of pixels correspondingto a second set of possible upper left corners.

The image analysis computer 110 recursively slides a localmaxima-locating window, e.g., an M×M window, where M is equal to orgreater than 3, over the second set of upper left pixels and eliminatesall pixels in each placement of the window except for the pixel having amaximum value such that a third set of upper left corner pixels isdefined.

The image analysis computer 110 determines pixel values corresponding toa first set of possible upper right corners of the one or more palletsusing a fourth corner template 520, the horizontal cross correlationresults and the vertical cross correlation results, see FIG. 7A. Thefourth corner template 520 is defined by a 3×3 window in the illustratedembodiment, corresponding to an upper right corner of a pallet.

The computer 110 slides the fourth corner template 520 concurrently overthe set of horizontal cross correlation results and the set of verticalcross correlation results so as to define pixel values for an upperright corner image. More specifically, the fourth corner template 520 ispositioned over the horizontal and vertical cross correlations so thatrequired values from each can be retrieved. Thus, a pixel location 521is moved sequentially over similarly positioned pixels in the first andsecond cross correlation images, see FIG. 7A. As the template pixellocation 521 is sequentially moved over each pixel in the first crosscorrelation image and a similarly located pixel in the second crosscorrelation image, a value for that pixel location is defined for theupper right corner image. During each placement of the fourth cornertemplate window 520, the computer 110 places into first and secondpixels 523A and 523B of an outer vertical column 524 correspondinghorizontal cross correlation results or pixel values, i.e., pixel valueshaving corresponding locations in the horizontal cross correlationimage, and places into first and second pixels 525A and 525B of a middlehorizontal row 526 corresponding vertical cross correlation results orpixels, i.e., pixel values having corresponding locations in thevertical cross correlation image. The computer 110 also negates thevalues added to the first and second pixels 523A, 523B and 525A, 525B inthe vertical column 524 and horizontal row 526. The computer 110 thendetermines the lowest or minimum value of pixels 523A, 523B, 525A and525B and assigns that lowest or minimum absolute value to the pixel 521of the vertical row 524, see FIG. 7A. The pixels in the resulting upperright corner image define a first set of pixels corresponding to a firstset of possible upper right corners.

The image analysis computer 110 compares the value of each pixel in thefirst set of upper right corner pixels to a fourth threshold value, suchas 0.5. All possible upper right corner pixels having a value less thanthe fourth threshold value are eliminated, i.e., the value for eacheliminated upper right corner pixel is changed to 0. All remaining upperright corner pixels, i.e., those having values equal to or greater thanthe fourth threshold value, define a second set of pixels correspondingto a second set of possible upper right corners.

The image analysis computer 110 recursively slides a localmaxima-locating window, e.g., an M×M window, where M is equal to orgreater than 3, over the second set of upper right pixels and eliminatesall pixels in each placement of the window except for the pixel having amaximum value such that a third set of upper right corner pixels isdefined.

In the above description of different image analysis and imagemanipulation steps, there are instances in which the columns of pixelsconsidered and used during the steps were limited to those columns ofpixels considered to be “between the forks.” In a similar manner, thenumber of rows of pixels considered and used during some of these stepsmay be reduced as well. For example, if the gray scale image has 752rows of pixels, then not all of these rows of pixels need to be usedand/or considered for all of the image analysis and manipulation stepspreviously described.

As one example, only 500 of the rows of pixels may be utilized and theother 252 rows of pixels ignored. If the fork carriage apparatus 40 ismoving upwards, then only the top 500 rows of pixels of the gray scaleimage may be utilized for corner finding and similar steps describedabove. Conversely, if the fork carriage apparatus 40 is travelingdownwards, then the bottom 500 rows of pixels of the image may beconsidered as relevant for analysis. Thus, the vehicle computer 50 canindicate to the image analysis computer 110 the direction of travel ofthe fork carriage apparatus 40. Using this information, the imageanalysis computer 110 can determine the vertical span of the rows ofpixels that are to be used or considered for various image analysis andmanipulation steps.

The corner finding process described above involved the normalized grayscale image which may be, for example, 480 (columns)×752 (rows) pixelsor, alternatively, involved a limited image area of the gray scale imagewhich may be, for example, 321 (columns)×752 (rows) pixels or even 321(columns)×500 (rows) pixels. Because many of the computations describedare performed once, or possibly twice, for each pixel, the computationalburden may be relatively high depending on the type of image analysiscomputer 110 being utilized. The computational steps used to evaluateand analyze the normalized gray scale image when attempting to identifypallet objects can be reduced by using a number of techniques. Inparticular, the concept of an image pyramid can be used to reduce thenumber of pixels in an image being analyzed at particular steps of theprocess of identifying pallet objects.

For example, if the normalized gray scale image includes 480×752 pixelsand is considered to be the base image of an image pyramid, then amid-level image can be defined having a coarser resolution, and atop-level image can be defined having an even coarser resolution. Forexample, a low-pass and sub-sampled mid-level image can be created byusing a 2×2 window to average four pixels of the base image in order togenerate a pixel value for one pixel in the mid-level image. Referringto FIG. 6F, a base image 720 is illustrated and a 2×2 window 724 is usedto select 4 pixels in a region of the base image 720. The values of thefour pixels in the window 724 can be averaged and that value placed inthe pixel 726 of the mid-level image 722. The result is that themid-level image 722 is created that has ¼ the number of pixels as thebase image. This same process can be performed on the mid-level image722 to create the top-level image of the image pyramid. This top levelimage will have 1/16^(th) the number of pixels as the base image. Usingthe example size described above for the base image, the mid-level imagewould be 240×376 pixels and the top-level image would be 120×188 pixels.

The previously described processes for creating a vertical crosscorrelation image and a horizontal cross correlation image can beapplied to the top-level image having the reduced number of pixels inorder to reduce the computational burden of locating a first set ofpossible left or right corners. As described earlier, the firststep-edge template, T(x), had a size of 12 elements and the secondstep-edge template, T(y), had a size of 24 elements. In applying theserespective templates to the top-level image, their sizes can be adjustedso that each template has 6 elements. Thus:T(x)_(top-level)=−1,−1,−1,+1,+1,+1T(y)_(top-level)=−1,−1,−1,+1,+1,+1

Using these templates, a horizontal cross correlation image and avertical cross correlation image are generated as described earlier suchthat the resulting images each have the same number of pixels as thetop-level image. The first corner template 300 is slid over the verticaland horizontal cross correlation images as before in order to calculateeach pixel value for a top-level lower-left corner image. As before, thenumber of pixels that will retain non-zero values in the top-levellower-left corner image can be reduced by zeroing out any pixels havinga value less than a threshold (e.g., 0.5) and using a 5×5, or other sizewindow to identify local maxima pixels in order to zero-out the otherpixels.

The pixels that remain in the top-level lower-left corner image (i.e.,the non-zero pixels) represent a set of pixels that may correspond to alower left corner of a pallet in the normalized gray scale image. Eachof these pixel locations from the top-level lower-left corner image isfurther analyzed to more concretely determine a possible location of alower left corner.

First, each pixel location in the top-level image is translated to acorresponding pixel location in the mid-level image. Although one pixellocation in a lower resolution image corresponds to four different pixellocations in a higher resolution image arranged in a 2×2 matrix, as the2×2 averaging is not perfect, artifacts remain, and a larger 6×6 searchregion is defined. The one pixel location identified in thelower-resolution image (e.g., the top-level image) allows the locationof the 36 pixels of the 6×6 search region to be calculated. For example,assuming the pixel coordinate system described earlier, the upper leftcorner of the 6×6 search region in the next level down in the pyramidcan be translated according to:x _(higher-res)=(x _(lower-res)−1)*2−2y _(higher-res)=(y _(lower-res)−1)*2−2

If the image portions of FIG. 6F are used, pixel location (53, 53) inthe lower resolution image is translated into pixel location (102, 102)in the higher resolution image which means the 6×6 search region 723 inthe higher resolution image includes the 36 pixels to the right, below,and inclusive of pixel (102, 102).

Each possible lower left corner location in the top-level lower leftcorner image can be translated into a 36 pixel search region in themid-level image. FIG. 6G illustrates an arbitrary 36 pixel search region732 in the mid-level image.

These 36 pixels in the set 732 will be used to once again createhorizontal and vertical cross correlation images, apply a cornertemplate 300, and then keeping only the maximum corner over the 6×6region. Because of the shape of the corner template (see FIG. 6), thetwo additional columns 736 to the left of the set 732 are needed forvertical correlations and the two additional rows 734 above the set 732are needed for horizontal correlations.

As before, a horizontal step-edge template is applied to the appropriatepixels to generate a horizontal cross-correlation image and a verticalstep-edge template is applied to the appropriate pixels to generate avertical cross-correlation image. In applying these respective templatesto pixels of the mid-level image, their size can be adjusted so thateach template has 12 elements. Thus:T(x)_(mid-level)=−1,−1,−1,−1,−1,−1,+1,+1,+1,+1,+1,+1T(y)_(mid-level)=−1,−1,−1,−1,−1,−1,+1,+1,+1,+1,+1,+1.

Because of the way the corner template 302 of FIG. 6 is constructed, notall of the 60 pixels of FIG. 6G require both a horizontalcross-correlation value and a vertical cross-correlation value to becalculated. For example, for pixel locations in the two rows of 734, avertical cross-correlation value is unnecessary and a verticalcross-correlation value from pixels in the right-most column 738 are notused either. Similarly, horizontal cross-correlation values are notneeded for the pixels in the bottom-most row 740 or the two columns 736.Thus, there are 42 horizontal cross-correlation values generated usingthe above template T(x)_(mid-level) and there are 42 vertical crosscorrelation values generated using the above template T(y)_(mid-level).The horizontal and vertical cross-correlation values are used in thesame manner as before, as described with respect to FIG. 6, in order tocalculate lower left corner values for the 36 pixel locations of the set732 of FIG. 6G.

Rather than locate more than one local maxima within the 36 pixels usinga sliding window technique, the one pixel from the 36 pixels having themaximum value is selected and all the other pixel locations are ignored.The identified pixel location in the mid-level image corresponds to apixel location in the base image that implicitly references 4 pixels ofthe base image.

The steps described above with respect to the translation from thetop-level image to the mid-level image are repeated but now are appliedto translating a pixel location from the mid-level image to the baseimage. Utilizing exactly the same steps as described above, a singlepixel location from the mid-level image is used to identify a singlepixel location in the base image that possibly corresponds to a lowerleft corner. One change that may be implemented is that when calculatingthe horizontal and vertical cross-correlation images based on the baseimage pixels, the original step-edge templates T(x) and T(y) can beused.

Thus, a possible lower left corner location was identified in thetop-level image. This pixel location was used to select a subset ofpixels in the mid-level image. This subset of pixels was furtheranalyzed to identify a pixel location in the base image for furtheranalysis.

The process described above is repeated for each of the possible lowerleft corners first identified in the top-level lower-left corner image.The result is a set of one or more pixel locations in the normalizedgray scale image that may correspond to a pallet lower left corner andis equivalent to and used in place of the third set of possible firstcorners. Furthermore, for each of these locations a lower left cornerscore Score_(LowerLeftCorner), discussed below, has been calculated andretained during the application of the mask of FIG. 6.

The above description of determining possible pallet lower left cornersinvolved using all the pixels in a normalized gray scale image havingdimension of 480×752 pixels. As noted previously, there can be a leftlimit and a right limit defined for x coordinate values such that onlypixels in a region of the gray scale image considered to be “between theforks” are used in such determinations. Similarly, there can be an upperlimit and a lower limit defined for y coordinate values such that onlypixels in a region near the top or bottom of the gray scale image areused in such determinations, as well.

The imaging camera 130 may, in some instances, comprise either a chargecoupled device (CCD) or complementary metal-oxide-semiconductor (CMOS)camera. Such cameras may have the capability to perform pixel binning.In a CCD camera, as light falls on each pixel, charge accumulates forthat pixel location. The amount of accumulated charge for a pixelcorresponds to that pixel's value and is represented by a color or grayscale level when the resulting image is displayed. With pixel binning,the CCD camera can combine the accumulated charge value for multiplepixels to arrive at a single pixel value for the group of pixels. Whilebinning allows increased sensitivity to light, the resolution of aresulting image is decreased.

Binning is normally referred to as 2×2 or 4×4 binning and this refers tothe horizontal and vertical dimensions of the bin size. A 2×2 binning,for instance, includes a square grouping of 4 pixels. In embodiments ofthe present invention, a CCD camera having a resolution of 480×752pixels can be utilized with a 2×2 pixel binning, for example, to acquireimages that are 240 (columns)×376 (rows) pixels in size. These imagescan be utilized with the techniques described above relating to applyingone or more templates to vertical and horizontal cross-correlationimages to identify possible lower left corners and other attributes ofpallets in the gray scale image. The technique of using an image pyramidfor analysis can also be used with these images having fewer pixels.However, it may be beneficial to only have two different image sizes inthe pyramid. For example, the base image may be 240×376 pixels in sizeand the top-level image may be 120×188 pixels in size.

However, when using images with fewer pixels some aspects of thetechniques described above may need to be modified. For example, if a240×376 pixel image size is being used, the columns considered to be“between the forks” can be defined with a different left limit value anda different right limit value. Also, the upper limit and lower limitvalues for what rows of the image to consider may be different as well.For example, for the smaller image size the left limit may be 40 and theright limit may be 200. Thus, only pixels located in columns 40 to 200would be considered “between the forks.” Also, if the fork carriageapparatus 40 is traveling upwards, then only the top 250 rows of theimage may be considered during analysis. Conversely, if the forkcarriage apparatus 40 is traveling downwards, then the bottom 250 rowsof the image may be considered during analysis.

A similar process can be used, but with the appropriate mask of FIG. 7,to identify a set of possible lower right corner locations in thenormalized gray scale image along with a corresponding lower rightcorner score Score_(LowerRightCorner), discussed below.

The process 232 for determining a first or left Ro image, which providesan orthogonal distance from an origin point to one or more possible leftvertical lines and one or more possible horizontal lines on one or morepossible pallets in a corresponding gray scale image and a second orright Ro image, which provides an orthogonal distance from an originpoint to one or more possible right vertical lines and one or morepossible horizontal lines on one or more possible pallets in acorresponding gray scale image, will now be described with reference toFIG. 8. While both the left and right Ro images may comprise pixelscorresponding to an offset from the origin of more or less horizontallines, the first Ro image may also comprise pixels corresponding to anoffset from the origin of left vertical lines, i.e., lines defining avertical interface between an open area on the left and a palletstructure on the right, which pixels have a value greater than 0. Thesecond Ro image may also comprise pixels corresponding to an offset fromthe origin of right vertical lines, i.e., lines defining a verticalinterface between a pallet structure on the left and an open area on theright, which pixels have a value greater than 0.

First the image analysis computer 110 determines a horizontal gradientimage, see step 260 in FIG. 8, by convolving gray scale image data froma given image frame with a first convolution kernel. The firstconvolution kernel G comprises in the illustrated embodiment:

G=0.0854, 0.3686, 0.8578, 0.9079, 0, −0.9079, −0.8578, −0.3686, −0.0854,wherein G is computed as a first derivative of a Gaussian function withits spread value chosen to deemphasize image noise. For example,choosing a spread value of σ=2 deemphasizes noise in the gray scaleimage. One example of a simplified Gaussian function shown below is theexponential of the negative square of the index x divided by the squareof the spread value. This simplified Gaussian function is then modifiedto produce a gradient kernel, G(x), such that the peak value of thekernel is +1 and its minimum value is −1.

Thus, starting with the following simplified Gaussian equation

${{Simplified}\mspace{14mu}{Gaussian}\mspace{14mu}(x)} = {\exp\left( {- \frac{x^{2}}{\sigma^{2}}} \right)}$

The first derivative over x is calculated as:

$\frac{{\mathbb{d}\mspace{14mu}{Simplified}}\mspace{14mu}{Gaussian}\mspace{14mu}(x)}{\mathbb{d}x} = {{- \frac{2x}{\sigma^{2}}}{\exp\left( {- \frac{x^{2}}{\sigma^{2\;}}} \right)}}$

The peak values of this derivative function occur at

$\pm \frac{\sigma}{\sqrt{2}}$

and the magnitude of the derivative function at these points is

$\frac{\sqrt{2}\;}{\sigma}{\exp(0.5)}$

By dividing the derivative function by this magnitude value, thederivative function can be modified so that its maximum value is +1 andits minimum value is −1. Performing the division generates the formulafor the gradient kernel G(x):

${G(x)} = {\frac{{- \sqrt{2}}x}{\sigma}{\exp\left( {0.5 - \frac{x^{2}}{\sigma^{2}}} \right)}}$

The computer 110 determines the horizontal gradient image by evaluatingthe following convolution equation using a first sliding window SW₁, seeFIG. 9, having a horizontal number of adjacent cells that is equal to anumber of elements of the first convolution kernel G, wherein the firstsliding window SW₁ is positioned so that each cell corresponds to arespective pixel location along a portion of a particular row of thenormalized gray scale image. The convolution equation for calculating avalue for each pixel of the horizontal gradient image comprises:

${g(k)} = {\sum\limits_{j}\left( {{u(j)}*{v\left( {k - j} \right)}} \right)}$wherein:

j=a particular pixel column location in the gray scale image wherein,for the convolution equation, a value of j ranges between a first pixelcolumn location in the portion of the particular row and a last pixelcolumn location in the portion of the particular row;

u(j)=pixel value in the portion of the particular row of the normalizedgray scale image having a j column location;

k=a pixel column location of the normalized gray scale image over whichthe first sliding window is centered;

(k−j)=is an index value for the first convolution kernel, as j varies inan increasing manner, this index value ranges from a highest index valuecorresponding to a last element of the first convolution kernel to alowest index value corresponding to a first element of the firstconvolution kernel; and

v(k−j)=a value of a particular element of the first convolution kernelat the index value (k−j).

With reference to FIG. 9, an example is provided of how the aboveconvolution equation is evaluated to find a column 100, row 1 pixelvalue in the horizontal gradient image. The sliding window SW₁ is showncentered at column location 100 in the normalized gray scale image andis positioned so that each cell of the sliding window corresponds to arespective pixel location along a portion, i.e., pixel column locations96-104 in the example of FIG. 9, of a particular row, i.e., row 1 in theexample of FIG. 9, of the normalized gray scale image. Hence, thesliding window SW₁ has pixel column location values extending from 96 to104. When the convolution equation is evaluated to find a column 101,row 1 pixel value, the sliding window SW₁ is shifted one pixel to theright such that it will then have pixel column location values extendingfrom 97 to 105. In the example illustrated in FIG. 9, the value for thepixel at column 100, row 1 in the horizontal gradient image wascalculated to equal 0.257.

The horizontal gradient image will have the same number of pixels as thecorresponding gray scale image such that there is a one-to-onecorrespondence between pixel locations in the gray scale image and thehorizontal gradient image. The pixels in the horizontal gradient imagemeasure a change in intensity at a corresponding location in the grayscale image when moving horizontally in the gray scale image and causedby a vertical interface between an open area and a pallet structure. Iffor example, 8-bits are being used to represent each pixel value, thenthe pixels in the horizontal gradient image may have values from about−2⁷ to about +2⁷. Depending on the number of bits allocated to representthe pixel values, the range of possible minimum and maximum values will,of course, vary. Values near zero correspond to a continuous area, i.e.,continuous pallet structure or open area; a vertical interface between apallet structure on the left and an open area on the right will resultin gradient values much less than zero and a vertical interface betweenan open area on the left and a pallet structure on the right will resultin gradient values much greater than zero. In FIG. 16D, a horizontalgradient image is shown corresponding to the gray scale image in FIG.16A. In FIG. 16D, a first area 1100 corresponds to pixel values near 0,a second area 1102 corresponds to pixel values much less than zero, athird area 1104 corresponds to pixel values much greater than zero and afourth area 1106 corresponds to pixel values substantially greater thanzero.

Next, the image analysis computer 110 determines a vertical gradientimage, see step 262 in FIG. 8, by convolving gray scale image data froma given image frame with a second convolution kernel, which, in theillustrated embodiment, is the same as the first convolution kernel. Asnoted above, the first convolution kernel G comprises:G=0.0854,0.3686,0.8578,0.9079,0,−0.9079,−0.8578,−0.3686,−0.0854

The computer 110 determines the vertical gradient image by evaluatingthe following convolution equation using a second sliding window havinga vertical number of adjacent cells that is equal to a number ofelements of the second convolution kernel, wherein the second slidingwindow is positioned so that each cell corresponds to a respective pixellocation along a portion of a particular column of the normalized grayscale image. The convolution equation for calculating a value for eachpixel of the vertical gradient image comprises:

${g(d)} = {\sum\limits_{c}\left( {{u(c)}*{v\left( {d - c} \right)}} \right)}$

wherein:

c=a particular pixel row location in the gray scale image wherein, forthe convolution equation, a value of c ranges between a first pixel rowlocation in the portion of the particular column and a last pixel rowlocation in the portion of the particular column;

u(c)=pixel value in the portion of the particular column of thenormalized gray scale image having a c row location;

d=a pixel row location of the normalized gray scale image over which thesecond sliding window is centered;

(d−c)=is an index value for the second convolution kernel, as c variesin an increasing manner, this index value ranges from a highest indexvalue corresponding to a last element of the second convolution kerneland a lowest index value corresponding to a first element of the secondconvolution kernel; and

v(d−c)=a value of a particular element of said second convolution kernelat the index value (d−c).

With reference to FIG. 10, an example is provided of how the aboveconvolution equation is evaluated to find a column 1, row 7 pixel valuein the vertical gradient image. The sliding window SW₂ is shown centeredat row location 7 in the gray scale image and is positioned so that eachcell of the sliding window corresponds to a respective pixel locationalong a portion, i.e., pixel row locations 3-11 in the example of FIG.10, of a particular column, i.e., column 1 in the example of FIG. 10, ofthe normalized gray scale image. Hence, the sliding window SW₂ has pixelrow location values extending from 3 to 11. When the convolutionequation is evaluated to find a column 1, row 8 pixel value, the slidingwindow SW is shifted down one pixel such that it will then have pixellocation values extending from 4 to 12. In the example illustrated inFIG. 10, the value for the pixel at column 1, row 7 in the verticalgradient image was calculated to equal −1.23.

The vertical gradient image will have the same number of pixels as thecorresponding gray scale image such that there is a one-to-onecorrespondence between pixel locations in the gray scale image and thevertical gradient image. The pixels in the vertical gradient imagemeasure a change in intensity at a corresponding location in the grayscale image when moving vertically from top to bottom in the gray scaleimage and caused by a horizontal interface between an open area andpallet structure. If for example, 8-bits are being used to representeach pixel value, then the pixels in the vertical gradient image mayhave values from about −2⁷ to about +2⁷. Values near zero correspond toa continuous area, i.e., continuous pallet structure or open area; ahorizontal interface between an upper pallet structure and a lower openarea will result in gradient values much less than zero and a horizontalinterface between an upper open area and a lower pallet structure willresult in gradient values much greater than zero. In FIG. 16E, avertical gradient image is shown corresponding to the gray scale imagein FIG. 16A. In FIG. 16E, a first area 1120 corresponds to pixel valuesnear 0, a second area 1122 corresponds to pixel values much less than 0,a third area 1124 corresponds to pixel values much greater than 0 and afourth area 1126 corresponds to pixel values substantially greater than0.

In determining the horizontal and vertical gradient images there will beinstances near the edges of the image that the sliding window will haveone or more cells that do not overlap an image pixel because the cellfalls outside the range of the image pixels. In these instances, thevalue of the image pixel when calculating the convolution equations willbe treated as equal to zero in order to minimize the effect that thesenon-existent pixels have on the convolution calculations.

After determining the horizontal and vertical gradient images, the imageanalysis computer 110 determines the first Ro image, ρ_(left), providingan orthogonal distance in pixels from an origin point on the normalizedgray scale image to one or more possible horizontal lines and leftvertical lines representative of one or more possible pallets in thegray scale image (that corresponds to the camera image), see step 264 inFIG. 8. The first Ro image is determined by the image analysis computer110 by evaluating the following equation:

${\rho_{left}\left( {x,y} \right)} = \frac{{xg}_{x} + {yg}_{y}}{\sqrt{g_{x}^{2} + g_{y}^{2}}}$

wherein:

x=a gradient pixel row location in a horizontal direction;

y=a gradient pixel column location in a vertical direction;

g_(x)=a gradient pixel value from the horizontal gradient imagecorresponding to pixel location (x, y); and

g_(y)=a gradient pixel value from the vertical gradient imagecorresponding to pixel location (x, y).

In a particular instance when both g_(x) and g_(y) equal zero for apixel, then the Ro image pixel value is set to zero.

The first Ro image will have the same number of pixels as thecorresponding gray scale image such that there is a one-to-onecorrespondence between pixel locations in the gray scale image and thefirst Ro image. The range of pixel values in the first Ro image may varybased on gray scale image dimensions, which, in the illustratedembodiment, are 480 pixels×752 pixels. Hence, the first Ro image pixels,which define an orthogonal distance from an origin point on thenormalized gray scale image to one or more possible horizontal lines andleft vertical lines, may have values from about −800 to about +800. Avertical line defining an interface between an open structure on theleft and a pallet structure on the right or a horizontal line definingan interface between an upper open area and a lower pallet structurewill be represented in the first Ro image by positive pixel values. Avertical line defining an interface between a pallet structure on theleft and an open area on the right or a horizontal line defining aninterface between a lower open area and an upper pallet structure willbe represented by negative values.

In FIG. 16F is a first Ro image corresponding to the gray scale image inFIG. 16A. In FIG. 16F, first regions 1110 corresponds to negative pixelvalues and second regions 1112 corresponds to positive pixel values. Afirst section 1112A of the second region 1112 corresponds to the leftsurface 208A of the center stringer 208 and a second section 1112B ofthe second region 1112 corresponds to the upper surface 200A of thebottom pallet board 200. FIG. 16F illustrates that first, or left, Roimage pixels will have a positive value when those pixel locations areover the left vertical surface 208A of the center stringer 208 or anupper horizontal surface 200A of the bottom pallet board 200. However,other pixels in the first Ro image may also have a positive value butnot necessarily be located over a left vertical surface of a centerstringer or an upper horizontal surface of a bottom pallet board.

Next, the image analysis computer 110 may determine a second Ro image,ρ_(right), providing an orthogonal distance in pixels from an originpoint on the normalized gray scale image (that corresponds to the cameraimage) to one or more possible horizontal lines and right vertical linesrepresentative of one or more possible pallets in the gray scale image,see step 266 in FIG. 8. The second Ro image is determined by the imageanalysis computer 110 by evaluating the following equation:

${\rho_{right}\left( {x,y} \right)} = \frac{{- {xg}_{x}} + {yg}_{y}}{\sqrt{g_{x}^{2} + g_{y}^{2}}}$

wherein:

x=a gradient pixel row location in a horizontal direction;

y=a gradient pixel column location in a vertical direction;

g_(x)=a gradient pixel value from the horizontal gradient imagecorresponding to pixel location (x, y); and

g_(y)=a gradient pixel value from the vertical gradient imagecorresponding to pixel location (x, y).

Similar to the first Ro image, the value of this equation is set to zeroif both g_(x) and g_(y) equal zero for a pixel.

The second Ro image will have the same number of pixels as thecorresponding gray scale image. The range of pixel values in the secondRo image may vary based on gray scale image dimensions, which, in theillustrated embodiment, are 480 pixels×752 pixels. Hence, the pixels mayhave values from about −800 to about +800. A vertical line defining aninterface between a pallet structure on the left and an open area on theright or a horizontal line defining an interface between an upper openarea and a lower pallet structure will be represented in the second Roimage by positive pixel values. A vertical line defining an interfacebetween an open structure on the left and a pallet structure on theright or a horizontal line defining an interface between a lower openarea and an upper pallet structure will be represented by negative pixelvalues.

In FIG. 16G a second Ro image is illustrated corresponding to the grayscale image in FIG. 16A. In FIG. 16G, first regions 1120 corresponds tonegative pixel values and second regions 1122 corresponds to positivepixel values. A first section 1122A of the second region 1122corresponds to the right surface 208B of the center stringer 208 and asecond section 1122B of the second region 1122 corresponds to the uppersurface 200A of the bottom pallet board 200. FIG. 16G illustrates thatsecond, or right, Ro image pixels will have a positive value when thosepixel locations are over the right vertical surface 208B of the centerstringer 208 or an upper horizontal surface 200A of the bottom palletboard 200. However, other pixels in the second Ro image may also have apositive value but not necessarily be located over a right verticalsurface of a center stringer or an upper horizontal surface of a bottompallet board.

Analysis of a single pixel in a Ro image does not definitively revealwhether or not that pixel is over a left (or right) vertical surface orupper horizontal surface of a pallet. Instead, an interface betweenpallet structure and an opening is characterized in a Ro image by astring of adjacent pixels having substantially the same pixel value,with that value being the orthogonal distance from the image origin tothat interface. Thus, a string of such pixels having nearly the samepositive Ro value and stretching from (x₁, y₁) to (x₂, y₂) in the Roimage represents an possible pallet line or edge in the gray scale imagethat also extends between (x₁, y₁) and (x₂, y₂). For a pallet edge thatextends substantially vertically, x₂ can equal x₁ and for asubstantially horizontal edge y₂ may equal y₁. Furthermore, the longerthe string of pixels is that share the same Ro value, the more likelythe string of pixels represents an actual edge of the pallet. Ideally,the pixels sharing the same Ro value will be directly adjacent to oneanother; however, in an actual image having noise and otherimperfections, “adjacent” can include pixels within a certain minimumdistance (e.g., ±5 pixels) of one another rather than only pixelsdirectly adjacent to one another.

An optional pre-qualification test in accordance with a first embodimentof the present invention for each lower left corner from the third setof lower left corners is defined by the process 234, which attempts totrace horizontal and vertical lines from each identified corner of thethird set of first possible corners, i.e., possible lower-left corners.The process 234 will now be described with reference to FIGS. 11A, 11B,12A, 12B, 13A and 13B. If a horizontal and a vertical line can be tracedfrom a possible lower-left corner, then that corner is consideredpre-qualified. If horizontal and vertical lines cannot be traced from apossible lower-left corner, then that possible corner isdisqualified/ignored. A traced horizontal line may correspond to anupper surface 200A of a bottom pallet board 200 and a traced verticalline may correspond to the left surface 208A of the pallet centerstringer 208, see FIG. 3.

A portion of an example first or left Ro image is illustrated in FIG.11A, which includes a possible first or lower left corner LLC shown at apixel location corresponding to image column 178, row 214.

A process for attempting to trace a horizontal line will now bedescribed with reference to the process steps illustrated in FIGS. 12Aand 12B.

After the image analysis computer 110 identifies one of the possiblelower left corners from the third set of lower left corners in the firstRo image, see step 322, it positions a J×K window 323 over at leastportions of a plurality of image rows in the first Ro image including atleast a portion of an image row containing the possible lower leftcorner LLC, see step 324 in FIG. 12A. More specifically, the window 323is centered over the row portion including the lower left corner LLC,with the column including the lower left corner defining the right-mostcolumn in the window 323. In the illustrated embodiment, the J×K window323 comprises a 10×7 window, see FIG. 11A.

The image analysis computer 110 then calculates an average pixel valuefor each row in the window 323, see step 326. In FIG. 11A, the window323 includes rows 211A-217A. Window rows 211A-217A define portions ofthe image rows 211-217 in the first Ro image in FIG. 11A. The averagepixel value or row mean for each window row 211A-217A is provided inFIG. 11A. For example, window row 211A has an average pixel value or rowmean of 217.50012.

The computer 110 then determines which window row 211A-217A in thewindow 323 has an average pixel value nearest the Y coordinate value ofa row Y_(T) of the current pixel location being considered for inclusionin a line being traced (a line being traced is also referred to hereinas “trace”), see step 328. In FIG. 11A, the lower left corner LLC atfirst Ro image column 178, row 214 defines the starting point of theline being traced and, hence, is the current pixel location beingconsidered for inclusion in the line being traced in the window 323. Tofind the window row having an average pixel value nearest the currentpixel location row Y_(T), 214 in the illustrated example, the computer110 first determines an absolute value of the difference |D| between thecurrent pixel location row Y_(T) and the row mean for each window row211A-217A using the following equation:|D|=|Y _(T)−row mean|  Equation 1:

|D| values for each window row 211A-217A are provided in FIG. 11A. Thecomputer 110 then identifies the window row 211A-217A having thesmallest absolute value of the difference between the current pixellocation row Y_(T) and its row mean. For the window 323 illustrated inFIG. 11A, window row 214A has an absolute value of the difference |D|between the current pixel location row Y_(T) and its row mean=0.01399,which is the smallest absolute value of the difference |D| of all of thewindow rows 211A-217A in the window 323, see FIG. 11A. The window rowhaving the smallest absolute value of the difference |D| is defined asthe “nearest window row”; hence, window row 214A is defined as thenearest window row in the window 323 of FIG. 11A.

The image analysis computer 110 then decides whether the nearest windowrow is over a pallet P, see step 330, by solving the following equation:

$\begin{matrix}\left\{ \begin{matrix}{{{{Y_{NWR} - {{row}\mspace{14mu}{mean}\mspace{14mu}({NWR})}}} \leq Y_{NWR}},\left. \Rightarrow{{over}\mspace{14mu} a\mspace{14mu}{pallet}} \right.} \\{{otherwise},\left. \Rightarrow{{not}\mspace{14mu}{over}\mspace{14mu} a\mspace{14mu}{pallet}} \right.}\end{matrix} \right. & {{Equation}\mspace{14mu} 2}\end{matrix}$

wherein: Y_(NRW)=Y coordinate value of an image row including thenearest window row in the J×K window; and

-   -   row mean (NWR)=average of pixel values in the nearest window row        in the J×K window.

In the illustrated example, Equation 2 is shown solved in FIG. 11A asbeing true, i.e., the nearest window row 214A is over a pallet P. If thenearest window row 214A is over a pallet, it is presumed that thecurrent pixel location, i.e., the possible lower left corner LLC atimage column 178, row 214, is also positioned over a pallet P. Thecomputer 110 then decides that the current pixel location (column 178,row 214) is to be included as one of the pixels that make up thehorizontal line, i.e., a possible bottom pallet board, being traced andincrements a horizontal line count by 1, see step 334. The computer 110then checks to determine if the count is equal to a predefined count,e.g., 100 in the illustrated embodiment, see step 336. If yes, then thecomputer 110 concludes that a horizontal line has been traced from thelower left corner LLC, see step 338. The computer 110 then goes to step370 in FIG. 13A, which will be discussed below, see step 340.

If the count is less than 100 in step 336, the computer 110 then stepsthe window 323 horizontally to the left by one pixel, centers thestepped window 323A over a new window row collinear with the image rowcontaining the previously determined nearest window row, row 214 in theexample illustrated in FIGS. 11A and 11B, and returns to step 326, seestep 342.

In FIG. 11B, the stepped window 323A is shown centered over row 214. Thewindow 323A comprises window rows 211B-217B. The current pixel locationin the stepped window 323A is at the right-most pixel of the centerwindow row in the window 323A, i.e., window row 214B. Hence, the currentpixel location being considered for inclusion in the line being tracedis at a pixel located at image column 177, row 214. The average pixelvalue for each of the window rows 211B-217B in the stepped window 323Ais provided in FIG. 11B. Window row 211B has an average pixel valuenearest the Y-coordinate value of row 214, which is the row of thecurrent trace location. Hence, window row 211B is the nearest window rowin the stepped window 323A. Equation 2 is shown solved in FIG. 211B astrue. As shown in FIG. 11B, solving Equation 2 above determines that thecurrent pixel location, i.e., the pixel at image column 177, row 214, ispositioned over a pallet P. The computer 110 then determines that thecurrent pixel location (column 177, row 214) should be included as apixel of the horizontal line, i.e., a possible bottom pallet board,being traced and increments a horizontal line count by 1, see step 334.The computer 110 then goes to step 336.

If, in step 330, it is decided that the nearest row in the window 323 isnot over a pallet, then the image analysis computer 110 centers an L×Mwindow 323B over the nearest row 214A in the window 323, see step 350,wherein the L×M window 323B is larger than the J×K window 323 tofacilitate capture of the pallet if possible. In the illustratedembodiment, the L×M window 323B comprises a 10×9 window. The computer110 then calculates an average of the pixel values for each row in theL×M window 323B, see step 352 and determines which row in the L×M window323B has an average pixel value nearest the Y coordinate value of thecurrent pixel location being considered for inclusion in the line beingtraced, which comprises the right-most pixel in the center row of thewindow 323 (the center row in the window 323B is the same as the nearestrow 214A in the window 323), see step 354. The row in the L×M windowhaving an average pixel value nearest the Y coordinate of the currentpixel location is defined as a “nearest row,” and the computer 110decides whether the nearest row in the L×M window 323B is over a palletusing Equation 2 set out above (wherein the L×M window is used whenevaluating Equation 2 instead of the J×K window), see step 356. If thenearest row in the L×M window 323B is determined to be over a pallet,the computer 110 sets the horizontal line count equal to 1, sets ano-horizontal line count equal to zero and then proceeds to step 342,see step 358. If the nearest row in the L×M window is determined not tobe over a pallet, then the no-horizontal line count is incremented by 1,see step 360. The computer 110 then determines if the no-horizontal linecount is equal to 20, see step 362. If yes, the lower left corner isdisqualified. If the no-horizontal line count is less than 20, then anL×M window is centered over the nearest row from the prior L×M window,see step 366, and the computer returns to step 352.

A further portion of the example first or left Ro image from FIG. 11A isillustrated in FIG. 11C, which includes the possible first or lower leftcorner LLC shown at a pixel location corresponding to image column 178,row 214.

A process for attempting to trace a vertical line will now be describedwith reference to the process steps illustrated in FIGS. 13A and 13B.After the image analysis computer 110 has traced a horizontal line fromthe possible lower left corner LLC, see steps 338 and 340, it positionsa P×Q window 371 over at least portions of a plurality of image columnsin the first Ro image including at least a portion of an image columncontaining the possible lower left corner LLC, see step 370 and FIG.11C. More specifically, the window 371 is centered over the columnportion including the lower left corner LLC, with the row including thelower left corner defining the lower-most row in the window 371. In theillustrated embodiment, the P×Q window 371 comprises a 5×6 window.

The image analysis computer 110 then calculates an average pixel valuefor each column in the window 371, see step 372. In FIG. 11C, the window371 includes columns 176A-180A. Window columns 176A-180A define portionsof the image columns 176-180 in FIG. 11C. The average pixel value orcolumn mean for each window column 176A-180A is provided in FIG. 11C.For example, window column 178A has an average pixel value or columnmean of 221.2977.

The computer 110 then determines which window column 176A-180A in thewindow 371 has an average pixel value nearest the X coordinate value ofa column X_(T) of the current pixel location being considered forinclusion in a line being traced, see step 374. In FIG. 11C, the lowerleft corner LLC at first Ro image column 178, row 214 defines thestarting point of the line being traced and, hence, is the current pixellocation being considered for inclusion in the line being traced in thewindow 371. To find the window column having an average pixel valuenearest the current pixel location column X_(T), the computer 110 firstdetermines an absolute value of the difference |D| between the currentpixel location column X_(T) and the column mean for each window column176A-180A using the following equation:|D|=|X _(T)−column mean|  Equation 3:

|D| values for each window column 176A-180A are provided in FIG. 11C.The computer 110 then identifies the window column 176A-180A having thesmallest absolute value of the difference between the current pixellocation column X_(T) and its column mean. For the window 371illustrated in FIG. 11C, window column 178A has an absolute value of thedifference |D| between the current pixel location column X_(T) and itscolumn mean=43.29768, which is the smallest absolute value of thedifference |D| of all of the window columns 176A-180A in the window 371,see FIG. 11C. The window column having the smallest absolute value ofthe difference |D| is defined as the “nearest window column”; hence,window column 178A is defined as the nearest window column in the window371 of FIG. 11C.

The image analysis computer 110 then decides whether the nearest columnis over a pallet P, see step 376, by solving the following Equation 4:

$\quad\left\{ \begin{matrix}{{\left. {X_{NWC} - {{column}\mspace{14mu}{mean}\mspace{14mu}({NWC})}} \middle| {\leq {1.5X_{NWC}}} \right.,\left. \Rightarrow{{over}\mspace{14mu} a\mspace{14mu}{pallet}} \right.}} \\{{otherwise},\left. \Rightarrow{{not}\mspace{14mu}{over}\mspace{14mu} a\mspace{14mu}{pallet}} \right.}\end{matrix} \right.$

wherein:

X_(NWC)=X coordinate value of a column in the Ro image including thenearest column from the P×Q window; and

-   -   column mean (NWC)=average of pixel values in the nearest window        column in the P×Q window.

In the illustrated example, Equation 4 is shown solved in FIG. 11C asbeing true, i.e., the nearest window column 178A is over a pallet. Ifthe nearest window column is over a pallet, it is presumed that thecurrent pixel location, i.e., the possible lower left corner LLC atimage column 178, row 214, is also positioned over a pallet P. Thecomputer 110 then decides that the current pixel location (column 178,row 214) in the nearest column 178A is to be included as one of thepixels that make up a vertical line, i.e., a possible left surface 208Aof a pallet center stringer 208, being traced and increments a verticalline or stringer count by 1, see step 380. The computer 110 then checksto determine if the stringer count is equal to a predefined count, e.g.,15 in the illustrated embodiment, see step 382. If yes, then thecomputer 110 concludes that a vertical line has been traced from thelower left corner LLC, see step 384. The computer 110 then validates thelower left corner LLC from which horizontal and vertical lines weretraced, see step 386 in FIG. 13A.

If the count is less than 15 in step 382, the computer 110 then stepsthe window 371 vertically upward by one pixel, centers the steppedwindow 371A over a new window column collinear with the image columncontaining the previously determined nearest window column, column 178Ain the example illustrated in FIGS. 11C and 11D, and returns to step372.

In FIG. 11D, the stepped window 371A is shown centered over column 178.The window 371A comprises window rows 176B-180B. The current pixellocation being considered for inclusion in the line being traced in thestepped window 371A is at the bottom-most pixel of the center windowcolumn in the window 371A, i.e., window column 178B. Hence, the currentpixel location being considered for inclusion in the line being tracedis at a pixel located at image column 178, row 213. The average pixelvalue for each of the window columns 176B-180B in the stepped window371A is provided in FIG. 11D. Window column 176B has an average pixelvalue nearest the X-coordinate value of column 178, which is the columnof the current pixel location. Hence, window column 176B is the nearestwindow column in the stepped window 371A. Equation 4 is shown solved inFIG. 11D as true. As shown in FIG. 11D, solving Equation 4 determinesthat the current pixel location, i.e., the pixel at image column 178,row 213, is positioned over a pallet P. Thus, computer 110 thendetermines that the current trace location (column 178, row 213) is apixel of a vertical line, i.e., a possible left surface 208A of a palletcenter stringer 208, to be traced and increments a vertical line countby 1, see step 380. The computer 110 then goes to step 382.

If, in step 376, it is decided that the one nearest column in the window371 is not over a pallet, then the image analysis computer 110 centersan R×S window 371B over the nearest column 178A in the window 371, seestep 390, to facilitate capture of the pallet if possible, wherein theR×S window 371B is larger than the P×Q window 371, see FIG. 11C. In theillustrated embodiment, the R×S window 371B comprises a 7×6 window. Thecomputer 110 then calculates an average of the pixel values for eachcolumn in the R×S window 371B, see step 392, determines which column inthe R×S window 371B has an average pixel value nearest the X coordinatevalue of the current pixel location being considered for inclusion inthe trace, which comprises the lower-most pixel in the center column ofthe window 371B (the center column in the window 371B is the same as thenearest column 178A in the window 371), see step 394, which column isdefined as a “nearest column” and decides whether the nearest column inthe R×S window 371B is over a pallet using Equation 4 set out above(wherein the R×S window is used when evaluating Equation 4 instead ofthe P×Q window), see step 396. If the nearest column in the R×S window371B is determined to be over a pallet, then the computer 110 sets thestringer count equal to 1, sets a no-vertical line count equal to zeroand then proceeds to step 388, see step 398. If the nearest column inthe R×S window is determined not to be over a pallet, then theno-vertical line count is incremented by 1, see step 400. The computer110 then determines if the no-vertical line count is equal to 20, seestep 402. If yes, then lower left corner LLC is disqualified. If theno-vertical line count is less than 20, then a R×S window is centeredover the nearest column from the prior R×S window, see step 406, and thecomputer returns to step 392.

A pre-qualification test in accordance with a second embodiment of thepresent invention for each lower left corner will now be described withreference to FIG. 14A-14D. The pre-qualification test involvesattempting to trace a horizontal line and a vertical line from eachpossible lower left corner. This test also uses the pixel values from aRo image and, particularly, pixel values from the first Ro image. If ahorizontal and a vertical line can be traced from a possible lower-leftcorner, then that corner is considered pre-qualified. If horizontal andvertical lines cannot be traced from a possible lower left corner, thenthat possible corner is disqualified/ignored. A traced horizontal linemay correspond to an upper surface 200A of a bottom pallet board 200 anda traced vertical line may correspond to the left surface 208A of thepallet center stringer 208, see FIG. 3.

A portion of an example first or left Ro image is illustrated in FIG.14A. A process for attempting to trace a horizontal line will now bedescribed with reference to the process steps illustrated in FIG. 14C.

After the image analysis computer 110 identifies one of the possiblelower left corners from the third set of lower left corners in the firstRo image, see step 602, it moves left a predetermined number of pixels,see step 604. For example, moving left by about 5 pixels is beneficialto increase the likelihood of starting the attempted tracing with anopening, rather than pallet structure, above the starting tracelocation. The image analysis computer 110 positions an N×O window 639over at least portions of a plurality of image rows in the first Roimage, see step 606 in FIG. 14C. In relation to FIG. 14A, a possiblelower left corner was identified to be at pixel location x=237, y=124which can also be referred to as X_(LLC)=237 and Y_(LLC)=124. Asmentioned, 5 pixels, for example, are subtracted from the x coordinatein step 604 to initialize the starting location 650 for the attemptedtrace. Thus, the starting location 650 for the example attempted traceis (232,124), see FIG. 14A. The N×O window 639 is positioned so that itis vertically centered over pixel location (232,124) and horizontallypositioned such that the right-most column overlays the pixel location(232,124). In the example in FIG. 14A, the N×O window 639 is a 13×8window, or matrix, and, by its positioning, there are 6 rows of thewindow 39 above the pixel location (232,124) and 6 rows below it. Alsothere are 7 columns of the window 639 to the left of the pixel location(232,124). Although a 13×8 window is shown as an example, other sizewindows can be used as well without departing from the scope of thepresent invention. Thus, the N×O window 639 extends in the y-directionfrom row 118 to row 130 and extends in the x direction from column 225to column 232. Each row is also referenced by an index value k, 646, asshown in FIG. 16A. The index value is assigned such that the top row ofthe N×O window 639 is at k=1 and the bottom row of the N×O window 639 isat k=13 (in this example).

As explained below, during each step of the tracing method newcoordinates (x_(new), y_(new)) are calculated for a subsequent step ofthe attempted tracing method to find a further point of the horizontalline being traced. When positioning the N×O window for a subsequentstep, the window is centered vertically over y_(new) and positionedhorizontally so that the right-most column is over x_(new).

In step 608, the image analysis computer 110 calculates an average W(k)(shown as column 652) for each of the rows of the N×O window 639. Theaverage is calculated using the pixel values from the first Ro image.Then, in step 610, the image analysis computer 110 determines the firstrow, starting with k=1, which satisfies three different tests:

-   -   1. W(k+1)>0 (shown as column 656);    -   2. MEAN(W(1:k))<0        -   where MEAN(W(1:k)) (shown as column 658)=

$\frac{1}{k}*{\sum\limits_{i = 1}^{k}{W(i)}}$

-   -   3. |Y_(LLC)−MEAN(W(k+1:13))|≦Y_(LLC) (shown as column 660)        -   where MEAN(W(k+1:13)) (shown as column 659)=

$\frac{1}{13 - k}*{\sum\limits_{j = {k + 1}}^{13}{W(j)}}$

If all the rows of the window 639 are tested and the image analysiscomputer 110 determines, in step 612, that no row is found thatsatisfies the above constraints, then the possible lower left corner isdiscarded and the process stops for this particular possible lower leftcorner. In the example window 639 of FIG. 14A, a successful row wasfound, and column 662 indicates that the first k value to satisfy theabove constraints is k=4 (shown as column 664). In step 614, the imageanalysis computer 110 uses this k value to identify the coordinates usedwhen determining how to position the N×O window 639 for the next step ofthe attempted tracing. In particular, in step 614, the image analysiscomputer 110 calculates a new x coordinate based onx_(new)=x_(current)−1. For the example of FIG. 14A, x_(new)=231. In step614, the image analysis computer also calculates a new y coordinatebased on y_(new)=y_(current)+k−6 (shown as column 666). For the exampleof FIG. 14A, y_(new)=122.

If the image analysis computer 110 determines, in step 616, that theconstraints of step 610 have been successfully met for the previous 100points/pixels of a traced line, then a horizontal line has beensuccessfully traced from the possible lower left corner. Processingstops for the horizontal trace and continues to the vertical tracingtechnique shown in FIG. 16D. If less than 100 points/pixels, or someother predetermined number of points/pixels, have been successfullytested, then tracing effort continues with step 606 using the new x andy coordinates (x_(new), y_(new)) to appropriately position the N×Owindow.

In the description of the process of FIG. 14C, step 610 provided a threeprong test to determine the row value “k” that would be used in step 614to position the N×O window 639 at its next location. As an alternative,a less restrictive and simpler test illustrated in FIG. 14E, could beused to determine the value “k”. After the image analysis computer 110,in step 608, calculates an average W(k) (shown as column 652) for eachof the rows of the N×O window 639. The image analysis computer 110,instead of the test of step 610 could alternatively, in step 610′,determine which average, W(k) minimizes |W(k)−Y_(LLC)| and then, in step612′, determine if the following inequality is true:|Y_(LLC)−W(k)|≦Y_(LLC) for that row, k. If the inequality is not true,then the image analysis computer 110 discards (237,124) as a possiblelower left corner; if the inequality is true, then new coordinates arecalculated according to step 614.

Using the alternative, simpler test of FIG. 14E, the row with k=6,having a W(k)=147.489988 satisfies the test of step 610′. That is, therow which has a k value that is equal to “6” has an average that isequal to “147.589988” and minimizes the value of |W(k)−Y_(LLC)| to beequal to “23.58999”. In addition, the inequality of step 612′ is alsosatisfied by this value of W(k) (i.e., |124−147.589988|) is less than orequal to 124.

As before, in step 614, the image analysis computer 110 uses theidentified k value to identify the coordinates used when determining howto position the N×O window 639 for the next step of the attemptedtracing. In particular, in step 614, the image analysis computer 110calculates a new x coordinate based on x_(new)=x_(current)−1. For theexample of FIG. 14A, x_(new)=231. In step 614, the image analysiscomputer also calculates a new y coordinate based ony_(new)=y_(current)+k−6. For the example of FIG. 14A, when using thealternative simpler test of FIG. 14E, y_(new)=124.

If a horizontal line is successfully traced from the possible lower leftcorner, then an attempt is made to trace a vertical line from thatpossible lower left corner as well.

A portion of an example first or left Ro image is illustrated in FIG.14B. A process for attempting to trace a vertical line will now bedescribed with reference to the process steps illustrated in FIG. 14D.In step 618, the image analysis computer 110 identifies the possiblelower left corner 678 as having coordinates (237,124) which makes the xcoordinate X_(LLC) equal to 237.

The image analysis computer 110 positions an V×W window 667 over atleast portions of a plurality of image columns in the first Ro image,see step 620 in FIG. 14D. As mentioned above, based on the coordinatesof the possible lower left corner, the starting location 678 for theexample attempted trace is (237,124). The V×W window 667 is positionedso that it is horizontally centered over pixel location (237,124) andvertically positioned such that the bottom-most row overlays the pixellocation (237,124). In the example in FIG. 14B, the V×W window 667 is a4×13 window, or matrix, and, by its positioning, there are 6 columns ofthe window 667 to the right of the pixel location (237,124) and 6columns to the left of it. Also there are 3 rows of the window 667 abovethe pixel location (237,124). Although a 4×13 window is shown as anexample, other size windows can be used as well without departing fromthe scope of the present invention. Thus, the V×W window 667 extends inthe y direction from row 121 to row 124 and extends in the x directionfrom column 231 to column 243. Each column is also referenced by anindex value k, 676, as shown in FIG. 14B. The index value is assignedsuch that the left-most column of the V×W window 667 is at k=1 and theright-most column of the V×W window 667 is at k=13 (in this example).

As explained below, during each step of the tracing method newcoordinates (x_(new), y_(new)) are calculated for a subsequent step ofthe attempted tracing method. When positioning the V×W window for asubsequent step, the window is centered horizontally over x_(new) andpositioned vertically so that the bottom-most row is over y_(new).

In step 622, the image analysis computer 110 calculates an average W(k)(shown as row 675) for each of the columns of the V×W window 667. Theaverage is calculated using the pixel values from the first Ro image.Next, in step 624, the image analysis computer 110 determines whichaverage, W(k) minimizes |W(k)−X_(LLC)| and determines, in step 626 ifthe following inequality is true: |X_(LLC)−W(k)|≦1.5 X_(LLC) for thatrow, k. If the inequality is not true, then the image analysis computer,in step 628, discards (237,124) as a possible lower left corner.

In the example of FIG. 14B, the column which has a k value that is equalto “5” has an average 681 that is equal to “234.5113” and minimizes thevalue 680 of |W(k)−X_(LLC)| to be equal to “2.4887”. As indicated by theanswer 684, the image analysis computer, in step 626 determined that theleft hand side of the inequality (i.e., |237−234.5113|) is less than orequal to 1.5×237 (i.e., 355.5). Thus, in step 630, the image analysiscomputer computes new x coordinates and y coordinates. In particular, instep 630, the image analysis computer 110 calculates a new y coordinatebased on y_(new)=y_(current)−1. For the example of FIG. 14B,y_(new)=123. In step 630, the image analysis computer also calculates anew x coordinate based on x_(new)=x_(current)+k−6. For the example ofFIG. 14B, x_(new)=236.

If the image analysis computer 110 determines, in step 632, that theconstraint of step 626 has been successfully met for the previous 10points/pixels of a traced line, then a vertical line has beensuccessfully traced from the possible lower left corner which is nowconsidered a prequalified possible lower left corner. If less than 10pixels, or some other predetermined number of points/pixels, have beensuccessfully tested, then tracing effort continues with step 620 usingthe new x and y coordinates (x_(new), y_(new)) to appropriately positionthe P×Q window.

After process 234 has been completed such that attempts have been madeto draw/trace a horizontal line and possibly a vertical line from eachidentified corner in the third set of possible first corners so as todefine a set of prequalified lower left corners, the image analysiscomputer 110 implements process 236. Hence, the computer 110 determines,for each prequalified lower left corner, which of a plurality ofpossible lines is most likely to be an actual line passing through theprequalified lower left corner. The process 236 will be discussed withreference to FIG. 15.

The above-described calculations involving the different Ro images andthe line tracing algorithms may benefit by considering the values of allthe pixel locations in the gray scale image. Thus, those particularimage analysis steps performed by the image analysis computer 110 arenot necessarily limited to being performed only on pixel locations thatare “between the forks.” However, once all the prequalified lower leftcorners have been identified, that set of pixel locations can be trimmedby discarding any prequalified lower left corner that is not located“between the forks.” Thus, in such an embodiment, none of the additionalanalysis and processing techniques described below are performed on anyprequalified lower left corner that has an x coordinate location outsideof the left limit and the right limit for x-coordinates (e.g., 80 and400, respectively).

The image analysis computer 110 defines a location of one of theprequalified lower left corners in the first Ro image, see step 410. Thecomputer 110 then defines a plurality of possible lines extendingthrough that prequalified lower left corner LLC, 61 lines in theillustrated embodiment, with each line being located at a correspondingangle φ with respect to a horizontal axis in the first Ro image, whereinthe range of angles is from −15 degrees to +15 degrees in 0.5 degreeincrements, see step 412 in FIG. 15. Two lines L₁ and L₂ are illustratedin FIG. 15A passing through the prequalified lower left LLC at pixellocation column 20, row 14. The remaining 59 possible lines are notillustrated in FIG. 15A. Line L₁ is at an angle φ₁=−10 degrees and lineL₂ is at an angle φ₂=10 degrees. As noted below, the computer 110determines which of the plurality of the possible lines is most likelyto be an actual line passing through the prequalified lower left cornerof a pallet, see step 414.

In order to define each of the plurality of possible lines passingthrough a given prequalified lower left corner LLC, the computer 110solves the following equation for each of the possible lines beingrespectively oriented at one angle so as to calculate an orthogonaldistance ρ(φ) from the origin of the Ro image to the line:ρ(φ)=−x _(PalletCorner) sin φ+y _(PalletCorner) cos φ  Equation 5:wherein:

ρ(φ)=an orthogonal distance from the origin point in the Ro image to apossible line at angle φ;

φ=an angle within the range of angles; and

(x_(PalletCorner),y_(PalletCorner))=coordinates in the Ro image for thepallet prequalified lower left corner.

In the illustrated embodiment, the equation ρ(φ) is evaluated 61 times,once for each of the 61 possible lines. In FIG. 15A, ρ(10) and ρ(−10)are illustrated.

After calculating ρ(φ) for a possible line, the computer 110 definespoints (e.g., pixel locations) for that line using the calculated valueof ρ(φ). Depending on the image size, each possible line may be definedby combining points from multiple sub-lines. The multiple sub-lines canbe considered as “thickening” the possible line so that a sufficientnumber of pixels can be included in the evaluation of the possible line.For example, if the image size is 480×752, then two sub-lines can beused to define the possible line. Alternatively, if the image size is240×376, then the possible line may be defined by only one sub-line.Thus, a possible line is defined by a set of sub-lines {L} according to:{L}={(1,y ₁ +k) to (M,y ₂ +k)}, where

-   -   k=0, 1, 2, . . . , (N_(y)−1),    -   N_(y)=the number of sub-lines used to define the possible line;        and

${{where}\text{:}\mspace{14mu} y_{1}} = {{round}\left( \frac{{{- {TOL}}\; 1} + {\rho(\varphi)} + {\sin\;\varphi}}{\cos\;\varphi} \right)}$${{and}\mspace{14mu} y_{2}} = {{round}\left( \frac{{{- {TOL}}\; 1} + {\rho(\varphi)} + {M\;\sin\;\varphi}}{\cos\;\varphi} \right)}$

-   -   given that:        -   ρ(φ)=an orthogonal distance from the origin point in the Ro            image to the corresponding possible line at angle φ            calculated from Equation 5 above;        -   φ=an angle within the range of angles;        -   TOL1=an empirically determined fixed value (equal to 0.5 in            the illustrated embodiment);        -   M=number of horizontal pixels in each row of the Ro image            (480 or 240 pixels in the illustrated embodiment); and        -   y₁=y or vertical coordinate value for the left end point of            the possible line if it extended, along the angle φ, to the            left-most image pixel column, x=1; and        -   y₂=y or vertical coordinate value for the right end point of            the possible line if it extended, along the angle φ, to the            right-most image pixel column x=M.

As mentioned, for a 240×376 size image, N_(y) may equal “1”; and for a480×752 size image, N_(y) may equal “2”. If N_(y)=“1”, then the set ofsub-lines {L} has only a single sub-line, L_(A), that has a left endpoint (1, y₁) and a right end point (M, y₂). Only points on thissub-line are used when evaluating the possible line. If, however, alarger image size were used (e.g., 480×752), then each possible line isdefined by combining points from a first sub-line L_(A) and a secondsub-line L_(B), in order to acquire a sufficient number of points.According to the above equation, the value k ranges from 0 to 1 andrespective end points for the two sub-lines L_(A) and L_(B) arecalculated such that:

the end points for the first sub-line L_(A) comprise:

-   -   (1,y₁) and (M,y₂); and

the end points for the second sub-line L_(B) comprise:

-   -   (1,y₁+1) and (M,y₂+1)

The computer 110 then generates first intermediate points between theend points for the first sub-line L_(A) and intermediate points betweenthe end points for the second sub-line L_(B) (if a second sub-line L_(B)is present). The respective intermediate points for both the first andsecond sub-lines L_(A) and L_(B) are determined using Bresenham's linealgorithm. As is well known to one of ordinary skill in the art, theBresenham line algorithm determines which points in a pixilated displayshould be selected in order to form a close approximation to a straightline between two given points. In practice, the algorithm can beefficiently implemented by calculating an error value for eachincremental step of the algorithm wherein the error value isrepresentative of the fractional portion of the actual y value at apoint x on the line. Accordingly, that error value is used to determinewhether to increase the y coordinate value or have it remain unchanged.Because, the error value at each step is an easily computed function ofthe previous error value and the slope of the line, the computationalexpense of implementing the algorithm can be minimal. The followingpseudo code effectively computes the intermediate points on thesub-lines with the constraint that the respective slope of each sub-lineis between −1 and +1; thus the pseudo code can be utilized for bothpositive sloped lines and negative sloped lines.

/* Draw a line from (x1, y1) to (x2, y2) It is assumed that x2 > x1 */error = 0 y = y1 deltax = x2 − x1 deltay = y2 − y1 If (deltax and deltayhave the same sign) Sign = 1 Else Sign = −1 For (x=x1 to x2){ Draw lineat (x,y) error = error + deltay If (2*sign*error >= deltax){ error =error − sign * deltax y = y + sign } }However, one of ordinary skill will recognize that if points on a linehaving a slope outside the range of −1 to +1 need to be calculated, thenthe above pseudo code can be easily adapted to perform such calculationsas well, without departing from the scope of the intended invention.

Next, the computer 110 evaluates the following expression for each pointon each possible line, i.e., this expression is evaluated for each pointon each sub-line (L_(A), L_(B), . . . ) corresponding to each possibleline:|ρ_(left)(X,Y)−ρ(φ)|≦TOL2*ρ(φ)  Equation 6:wherein:

(X,Y)=one of the points on a possible line;

ρ_(left)(X,Y)=value from the first Ro image at a point X,Y on thepossible line;

ρ(φ)=an orthogonal distance from the origin point in the Ro image to thepossible line at angle φ calculated from Equation 5 above;

φ=an angle within the range of angles; and

TOL2=an empirically determined fixed value (equal to 0.15 in theillustrated embodiment).

A respective counter, Accumulator Count(φ), may be associated with thepossible line having the angle φ, and can be incremented each time theexpression of Equation 6 is true for a point X, Y on the possible linehaving the angle φ. Thus, there are 61 counters, each having arespective value. As mentioned above, in some instances, to get asufficient number of points, more than one sub-line may be associatedwith a counter. After all points on all 61 possible lines have beenevaluated using Equation 6, the computer 110 identifies the line havingthe greatest count value as the line most likely to be an actual linepassing through the prequalified pallet left corner LLC, hereinafterreferred to as a bottom pallet board line. The bottom pallet board lineis assigned a confidence score, Score_(BaseboardLine), based on themaximum value of the 61 counters, according to the following equation:

${Score}_{BaseboardLine} = {\max\left( \frac{{AccumulatorCount}(\varphi)}{M \times N_{y}} \right)}$

wherein:

-   -   M=a number of horizontal pixels in each row of the Ro image (480        or 240 pixels in the illustrated embodiments); and    -   N_(y)=the number of sub-lines used to define each possible line.

Once the computer 110 identifies the line most likely to be an actualline passing through the prequalified pallet lower left corner LLC, thisline is presumed to represent a possible upper surface 200A of a bottomor lower pallet board 200 associated with this particular prequalifiedpallet lower left corner LLC. The location of the prequalified lowerleft corner LLC and the possible upper surface of the bottom palletboard can then be used in process 238, noted above, which attempts toidentify a set of possible lower right corners corresponding to theprequalified lower left corner. The set of possible lower right cornerswill satisfy certain criteria with respect to the prequalified lowerleft corner and the corresponding bottom pallet board upper surface,which are as follows:

-   -   1) A corresponding lower right corner will likely be located        within a certain horizontal range of the prequalified possible        lower left corner LLC. Hence, the computer 110 reviews all of        the possible lower right corners as defined by the third set of        possible second corners and discards those located closer than a        minimum horizontal distance (x_(min)) to the prequalified lower        left corner LLC or farther than a maximum horizontal distance        (x_(max)) from the prequalified lower left corner LLC. The        particular x_(min) and x_(max) values are measured in “pixels”        and depend on the range of center pallet stringer sizes being        visualized and how far the rack R is from the vehicle 10. For        typical rack to vehicle distances of from about 50 inches to        about 60 inches and pallets made of wood or a polymeric material        having a center stringer with a width W of from about 2 inches        to about 12 inches, x_(min) may equal 5 pixels and x_(max) may        equal 150 pixels, wherein the field of view is 480 pixels×752        pixels, see FIG. 17.    -   2) A corresponding lower right corner will likely be located        within a certain vertical distance, or skew, defining a vertical        range of +/−Y_(tol) from the prequalified lower left corner LLC.        Thus, the computer 110 discards any possible lower right corner        that falls outside of a vertical skew as defined by +/−Y_(tol),        which, for example, may equal +/−10 pixels, see FIG. 17.    -   3) An actual lower right corner will likely be located within a        certain vertical distance range from the bottom pallet board        line. Thus, the computer 110 discards any possible lower right        corner that falls outside a distance, +H_(max), above the bottom        pallet board line or extends beyond a distance, −H_(max), below        the bottom pallet board line. For example, +H_(max) may equal 4        pixels above the bottom pallet board upper surface and −H_(max)        may equal 4 pixels below the bottom pallet board line, see FIG.        17.

The location criteria associated with locating possible lower rightcorners may be adjusted according to the image size. For example, thevalues provided above may be appropriate for images which are 480×752pixels in size. However, for images which are 240×376 pixels in size,the values may be adjusted so that each is approximately half of theexample values provided above. The vertical skew, for example, may equal+/−5 pixels, the value for +H_(max) may equal 2 pixels above the bottompallet board surface and the value for −H_(max) may equal 2 pixels belowthe bottom pallet board line. Similarly, x_(mm) may equal 2 pixels andx_(max) may equal between 50 to 75 pixels.

A single prequalified lower left corner LLC and a corresponding bottompallet board line may be paired with one or multiple possible lowerright corners defining one or multiple candidate objects. A “candidateobject” is also referred to herein as a “possible pallet object.”

In the further embodiment noted above, where the imaging camera islocated above the forks 42A, 42B looking down, one or multiple possibleupper right corners are paired with each prequalified possible upperleft corner, wherein the limits on the search for the upper rightcorners are similar to those for the lower right corners.

For each candidate object comprising a prequalified lower left cornerLLC, a corresponding bottom pallet board line and a correspondingpossible lower right corner, the image analysis computer 110 attempts toidentify a distance between a point on the upper pallet board 202 andthe prequalified lower left corner LLC such that a pallet hole orfork-receiving opening lies in between. Using the first Ro image, theimage analysis computer 110 steps horizontally left from theprequalified lower left corner LLC a predetermined number of pixels(e.g., 10 pixels) and moves upward until the pixels transition frompositive to negative to positive. The step to the left ensures thetraversal upward occurs in a region where there is an opening ratherthan only pallet structure above the starting point. Thepositive-to-negative-to-positive transition estimates the location of apotential point on the upper pallet board 202 relative to the uppersurface 200A of the bottom pallet board 200, wherein the distancebetween the point on the upper pallet board 202 and the upper surface200A of the bottom pallet board 200 is a rough estimate of the distancebetween a lower surface 202A of the upper pallet board 202 and the uppersurface 200A of the bottom pallet board 200, see FIG. 3. Additionally,another method of estimating the distance from the bottom pallet boardupper surface to a possible point on the upper pallet board 202 isperformed by the computer 110 by moving upward from the prequalifiedlower left corner LLC until a negative pixel value in the first Ro imageis encountered. A negative pixel is encountered when a positive verticalsequence expires before reaching the upper pallet board 202, such as dueto noise or a lighting variation, or the positive vertical sequenceexpires upon reaching the upper pallet board 202. The maximum of thesetwo estimates is selected as a determination of the distance from theprequalified lower left corner to the point on the upper pallet board202.

The distance from the prequalified lower left corner/upper surface 200Aof the bottom pallet board 200 to the point on the upper pallet board202 is then used to roughly estimate the locations of the possible upperleft corner and the possible upper right corner. The possible upper leftcorner is presumed located in alignment with and above the prequalifiedlower left corner by a distance equal to the distance from theprequalified lower left corner to the point on the upper pallet board202. The possible upper right corner is presumed located in alignmentwith and above the possible lower right corner by a distance equal tothe distance from the prequalified lower left corner to the point on theupper pallet board 202.

For each candidate object comprising a prequalified lower left corner214, a corresponding bottom pallet board line (i.e., a possible uppersurface 200A of a bottom pallet board 200) and a corresponding possiblelower right corner, and for which locations of corresponding upper leftand right corners were estimated, possible left and right verticalstringer edges or lines may be calculated according to a methoddescribed below. However, an alternative method is also presentedfurther below that does not explicitly calculate both left and rightstringer edges.

For a potential left vertical stringer line, an orthogonal distance,ρ_(1desired), and an angle, φ₁, are calculated from Equations (7a and7b) below:

$\begin{matrix}{\varphi_{1} = {\tan^{- 1}\left( \frac{{P_{LowerLeft}(y)} - {P_{UpperLeft}(y)}}{{P_{LowerLeft}(x)} - {P_{UpperLeft}(x)}} \right)}} & {{Equation}\mspace{14mu} 7a} \\{\rho_{1{desired}} = {{{- {P_{LowerLeft}(x)}}\sin\;\varphi_{1}} + {{P_{LowerLeft}(y)}\cos\;\varphi_{1}}}} & {{Equation}\mspace{14mu} 7b}\end{matrix}$

where:

φ₁=is an angle between a horizontal line and a potential left stringerline, wherein the potential left stringer line extends from theprequalified lower left corner to the estimated upper left corner;

P_(LowerLeft)(y)=the Y coordinate value for the prequalified lower leftcorner;

P_(UpperLeft)(y)=the Y coordinate value for the estimated upper leftcorner;

P_(LowerLeft)(x)=the X coordinate value for the prequalified lower leftcorner;

P_(UpperLeft)(x)=the X coordinate value for the estimated upper leftcorner; and

ρ_(1desired)=the orthogonal distance from the origin point on thenormalized gray scale image to a vertical line passing through theprequalified lower left corner and the upper left corner.

For a potential right vertical stringer line, an orthogonal distance,ρ_(2desired), and an angle, φ₂, are calculated from Equations (8a and8b):

$\begin{matrix}{\varphi_{2} = {\tan^{- 1}\left( \frac{{P_{LowerRight}(y)} - {P_{UppperRight}(y)}}{{P_{LowerRight}(x)} - {P_{UpperRight}(x)}} \right)}} & {{Equation}\mspace{14mu} 8a} \\{\rho_{2{desired}} = {{{- {P_{LowerRight}(x)}}\;\sin\;\varphi_{2}} + {{P_{LowerRight}(y)}\cos\;\varphi_{2}}}} & {{Equation}\mspace{14mu} 8b}\end{matrix}$where

φ₂=is an angle between a horizontal line and a potential right stringerline, wherein the potential right stringer line extends from the lowerright corner to the estimated upper right corner;

P_(LowerRight)(y)=the Y coordinate value for the lower right corner;

P_(UpperRight)(y)=the Y coordinate value for the estimated upper rightcorner;

P_(LowerRight)(x)=the X coordinate value for the lower right corner;

P_(UpperRight)(x)=the X coordinate value for the estimated upper rightcorner; and

ρ_(2desired)=the orthogonal distance from an origin point on thenormalized gray scale image to a vertical line passing through the lowerright corner and the upper right corner.

FIG. 19A is illustrative of using the above equations to calculatevalues that represent a potential left stringer line or a potentialright stringer line. FIG. 19A does not represent actual pixel values ofany of the images previously discussed (e.g., gray scale image, Roimage, etc.) but, instead provides a framework to help understand thegeometry used for Equations 7a, 7b, 8a and 8b. However, one correlationwith the earlier discussed figures is that the (x, y) coordinates of thedifferent points in FIG. 19A, as well as the origin, do correspond tothe coordinate system and origin of the gray scale image. Theprequalified lower left corner is located at point 1906 in FIG. 19A andthe corresponding upper left corner location estimated earlier islocated at point 1908. The estimated upper left corner is presumed to belocated horizontally in line with the prequalified lower left corner,such that they have identical X coordinates, but is shown slightlyoffset from the prequalified lower left corner in FIG. 19A. Thepotential stringer line 1912 extends between the two corners 1906, 1908.The potential left stringer line 1912 can be characterized by a pair ofvalues (ρ_(1desired), θ), where, as shown in FIG. 19A, ρ_(1desired) isdesignated by line 1904 which represents the orthogonal distance of theline 1912 from the origin 1914, and the angle θ is designated by theangle 1902 which is the angle of line 1904 relative to the x-axis 1916.The parameter φ₁ is designated by the angle 1910 and is the angle formedby the potential left stringer line 1912 and a horizontal line 1918.Because θ=π/2+φ₁, the orthogonal distance ρ_(1desired) can be determinedaccording to Equation 7b above. Although the above discussion describingFIG. 19A relates to a potential left stringer line, a similar approachcan be used when solving Equations 8a and 8b for a potential rightstringer line.

With the angle ρ₁ (1910) calculated using Equation 7a above, it can thenbe determined whether a first set of pixels defining a left verticalline above the prequalified lower left corner from the first Ro imagematch the calculated orthogonal distance ρ_(1desired) by using pixelvalues from the first Ro image that are vertically in-line with, andextend upward from above the prequalified lower left corner to theestimated upper left corner. Using these pixels, a score valueScore_(left) is calculated that represents a value that is related tothe likelihood that a potential left vertical stringer line is actuallya left edge of a pallet stringer in the gray scale image. One method forcalculating this score value, Score_(left), is provided with referenceto FIG. 20A and FIG. 20B. FIG. 20A depicts a flowchart for scoring apotential left stringer line for a prequalified lower left corner andFIG. 20B depicts a region of pixels from the first Ro image located nearthe prequalified lower left corner. For the prequalified lower leftcorner 2030, see FIG. 20B, the vertical line being calculated representsa possible left vertical edge of the stringer and, thus, values from thefirst or left Ro image are used. For example, a set of Ro image valuesare selected above the prequalified lower left corner; the set can belimited to a maximum number of pixels such as ten pixels. In otherwords, for the prequalified lower left corner, the number of pixels tobe selected from the left Ro image is determined in step 2002, usingEquation 9a:h _(left)=min(P _(LowerLeft)(y)−P _(UpperLeft)(y),10)

According to the above equation, one of two values is used to determinethe number of pixels to be selected; either a height of the stringer inpixels (i.e., P_(LowerLeft)(y)−P_(UpperLeft)(y)) or 10 pixels, whichevervalue is lower. Thus, in step 2004, a set 2032, see FIG. 20B, of pixelsis selected from the left Ro image, wherein the set of pixels hash_(left) members in the set. Each member is a pixel having an (x,y)coordinate and a pixel value. As an example, if the prequalified lowerleft corner 2030 is located at (x, 225) and the estimated upper leftcorner 2036 was located at (x, 219), then h_(left)=6 and the members ofthe set 2032 of pixels selected are shown with hash marks in FIG. 20B.

The image analysis computer 110, in step 2006, initializes anaccumulator A₁ and an index value i, for later calculation steps. Theindex value i according to a particular example of the flowchart in FIG.20A ranges from 1 to h_(left) and is used to uniquely designate arespective member of the set of pixels 2032. The calculations involvingthe pixels values within the set 2032 of pixels are not directiondependent and, thus, the lowest index value can refer to the top-mostpixel 2036 of the set 2032 and increase while moving down the settowards the bottom-most pixel 2034 of the set 2032. However, the indexvalue can be reversed without affecting the calculations describedbelow. The index value i is tested by the image analysis computer 110 atstep 2008 and if the index value indicates that there are more pixels inthe set 2032 left to evaluate, then the process moves to step 2010. Atstep 2010, the image analysis computer 110 determines if the followinginequality is true:|ρ(x _(i) ,y _(i))−ρ_(1desired) |≦TOL4*,ρ_(1desired)

wherein:

i=the index value referring to a respective member of the first set ofpixels 2032;

ρ(x_(i),y_(i))=a value of the i^(th) pixel taken from the first set 2032of pixels in the first Ro image located directly above the prequalifiedlower left corner 2030;

ρ_(1desired)=the orthogonal distance from the origin point on thenormalized gray scale image to a line passing through the prequalifiedlower left corner and the upper left corner which was calculated abovewith respect to Equation 7b; and

TOL4=an empirically determined value (equal to 0.5 in the illustratedembodiment).

If the inequality is true for the particular, indexed pixel in step2010, then the accumulator A₁ is incremented, in step 2014, for this set2032 of pixels and the index value is incremented in step 2012 to thenrepeat the steps starting with step 2008. If the inequality is not truein step 2010, then the index value is incremented in step 2012 to thenrepeat the steps starting with 2008. Once the image analysis computer110 determines, in step 2008, that all the members of the set 2032 ofpixels have been evaluated, a left vertical line score value iscalculated, in step 2016, which represents the confidence level ofwhether or not the selected pixels 2032 located above the prequalifiedlower left corner 2030 represent an actual stringer left edge. The leftvertical line score value is the ratio of the respective accumulator bincount A₁ to the total count of the pixels selected for a particularline:

$\begin{matrix}{{Score}_{left} = \frac{A_{1}}{h_{left}}} & {{Equation}\mspace{14mu} 9b}\end{matrix}$

A process similar to that of FIG. 20A can be used to determine a rightvertical line score value as well. With the angle φ₂ calculated usingEquation 8a above, it can then be determined whether a second set ofpixels defining a right vertical line above a possible lower rightcorner from the second or right Ro image matches the calculatedorthogonal distance ρ_(2desirec) by using the pixel values from thesecond Ro image that extend from above the possible lower right cornerupwards to the estimated upper right corner. For the possible lowerright corner, the vertical line being calculated represents a possibleright vertical edge of the stringer and, thus, values from the second orright Ro image are used. For example, a set of Ro image values areselected above the possible lower right corner; the set can be limitedto a maximum number of pixels such as ten pixels. In other words, abovethe possible lower right corner, the number of pixels selected from theright Ro image are:h _(right)=min(P _(LowerRight)(y)−P _(UpperRight)(y),10)  Equation 10a:

Thus, starting at the possible lower right corner, the h_(right) numbersof pixels are selected that are located above the possible lower rightcorner.

A second accumulator bin A₂ is incremented by one if the followingexpression is true for a pixel from second set of pixels:|ρ(x _(j) ,y _(j))−ρ_(2desired) |≦TOL4*ρ_(2desired)

wherein:

j=the index value referring to a respective member of the second set ofpixels;

ρ(x_(j), y_(j))=a value of the j^(th) pixel taken from the second set ofpixels in the second Ro image located directly above the lower rightcorner;

ρ_(2desired)=the orthogonal distance from the origin point on thenormalized gray scale image to a line passing through the possible lowerright corner and the upper right corner which was calculated above withrespect to Equation 8b;

TOL4=an empirically determined value (equal to 0.5 in the illustratedembodiment).

A right vertical line score value is calculated which represents theconfidence level of whether or not the selected pixels located above thelower right corner represent an actual stringer right edge. The rightvertical line score value is the ratio of the respective accumulator bincount A₂ to the total count of the pixels selected for a particularline:

$\begin{matrix}{{Score}_{right} = \frac{A_{2}}{h_{right}}} & {{Equation}\mspace{14mu} 10b}\end{matrix}$

A candidate object that was potentially thought to include a centerstringer is rejected if either the left or the right score is less thana threshold, 0.1 in the preferred embodiment, or the sum of theScore_(left) and Score_(right) is less than a second threshold, 0.5 inthe preferred embodiment.

As mentioned above, an alternative method of evaluating a possiblestringer is also contemplated. FIG. 19B provides a geometrical frameworkhelpful in describing this alternative embodiment. As before, aprequalified lower left corner 1926 and a bottom pallet board line 1930are identified. As described above, these possible pallet objectfeatures are used to determine a possible lower right corner 1936, anestimated upper left corner 1928, and an estimated upper right corner1938. In particular, as defined above, the following values aredetermined by utilizing the assumptions that a) the prequalified lowerleft corner 1926 and the estimated upper left corner 1928 have the samex coordinate; and b) the possible lower right corner 1936 and theestimated upper right corner 1938 have the same x coordinate:

P_(LowerLeft)(y)=the Y coordinate value for the prequalified lower leftcorner 1926;

P_(UpperLeft)(y)=the Y coordinate value for the estimated upper leftcorner 1928;

P_(LowerLeft)(x)=the X coordinate value for the prequalified lower leftcorner 1926;

P_(UpperLeft)(x)=the X coordinate value for the estimated upper leftcorner 1928;

P_(LowerRight)(y)=the Y coordinate value for the possible lower rightcorner 1936;

P_(UpperRight)(y)=the Y coordinate value for the estimated upper rightcorner 1938;

P_(LowerRight)(x)=the X coordinate value for the possible lower rightcorner 1936; and

P_(UpperRight)(x)=the X coordinate value for the estimated upper rightcorner 1938.

Using these coordinate values along with the pixel values of thehorizontal gradient image described above, an alternative Score_(left)and Score_(right) can be calculated instead of those described earlier.In particular, the alternative Score_(left) value can be calculatedaccording to:

${Score}_{left} = {\frac{1}{{P_{LowerLeft}(y)} - {P_{UpperLeft}(y)}}{\sum\limits_{k = 1}^{({{P_{LowerLeft}{(y)}} - {P_{UpperLeft}{(y)}}})}{g_{x}\left( {{P_{LowerLeft}(x)},\left( {{P_{LowerLeft}(y)} - k} \right)} \right)}}}$where:

P_(LowerLeft)(y)=the Y coordinate value for a prequalified lower leftcorner;

P_(UpperLeft)(y)=the Y coordinate value for an estimated upper leftcorner;

P_(LowerLeft)(x)=the X coordinate value for the prequalified lower leftcorner;

k=an index value for the summation, wherein k varies from 1 to a maximumvalue of P_(LowerLeft)(y)−P_(UpperLeft)(y); and

g_(x)(P_(LowerLeft)(x),(P_(LowerLeft)(y)−k))=the pixel value from thehorizontal gradient image for the pixel having an x-coordinate ofP_(LowerLeft)(x) and a y-coordinate of (P_(LowerLeft)(y)−k).

A candidate object that was potentially thought to include a centerstringer may be rejected if its left stringer score, Score_(left), isless than a threshold. For example, in this alternative embodiment, thethreshold value for the alternative Score_(left) may be 5.0.

Additionally, an alternative Score_(right) may also be calculated andused in conjunction with the alternative Score_(left) value to determinewhether or not to reject a candidate object. In particular, thealternative Score_(right) value can be calculated according to:

${Score}_{right} = {\frac{1}{{P_{LowerRight}(y)} - {P_{UpperRight}(y)}}{\sum\limits_{k = 1}^{({{P_{LowerRight}{(y)}} - {P_{UpperRight}{(y)}}})}{g_{x}\left( {{P_{LowerRight}(x)},\left( {{P_{LowerRight}(y)} - k} \right)} \right)}}}$

where:

P_(LowerRight)(y)=the Y coordinate value for a possible lower rightcorner;

P_(UpperRight)(y)=the Y coordinate value for an estimated upper rightcorner;

P_(LowerRight)(x)=the X coordinate value for the possible lower rightcorner;

k=an index value for the summation, wherein k varies from 1 to a maximumvalue of P_(LowerRight)(y)−P_(UpperRight)(y); and

P_(UpperRight)(y) the pixel value from the horizontal gradient image forthe pixel having an x-coordinate of P_(LowerRight)(x) and a y-coordinateof (P_(LowerRight)(y)−k)

As before, a candidate object that was potentially thought to include acenter stringer may be rejected if its right stringer score,Score_(right), is less than a threshold. For example, in thisalternative embodiment, the threshold value for Score_(right) may be thesame as for the left vertical stringer for the alternative Score_(left)(e.g., 5.0). If either Score_(right) or Score_(left) are below theirrespective threshold values, or their sum is below some other threshold,then the candidate object may be rejected from further consideration.

Additionally, a candidate object that was potentially thought to includea center stringer may be deleted if its hotness is too great. In FIG.19B a region of pixels that might be a center stringer are defined bythe prequalified lower left corner 1926, the possible lower right corner1936, the estimated upper right corner 1938 and the estimated upper leftcorner 1928. If the variance of the pixel values in the normalized grayscale image for this region of pixels is greater than a predeterminedthreshold, then this candidate object may be rejected. An example is anobject produced by imaging a ceiling light. Ceiling lamps may emit nearinfrared illumination and show up as very bright irregular regions on adark background giving a high hotness level. The predetermined hotnessthreshold may, for example, be about 1200.

A variance, H_(var), for this potential center stringer region can becalculated according to:

$H_{v\;{ar}} = {\frac{1}{k}{\sum\limits_{i = 1}^{k}\left( {{{GSPV}\left( {{pixel}(i)} \right)} - \overset{\_}{GSPV}} \right)^{2}}}$

where:

-   -   k=the number of pixels in the region bounded by the corners        1926, 1928, 1936, and 1938;    -   i=an index value uniquely referring to one of the k pixels in        the region bounded by the corners 1926, 1928, 1936, and 1938;    -   GSPV(pixel(i))=a gray scale pixel value (GSPV) of a particular        one of the k pixels (i.e, pixel(i)) from the normalized gray        scale image in the region bounded by the corners 1926, 1928,        1936, and 1938; and    -   GSPV=an average of all the k pixels' gray scale pixel values        pixel from the normalized gray scale image in the region bounded        by the corners 1926, 1928, 1936, and 1938.

A further criterion involves the object area of the region that may be acenter stringer of a candidate object. If the region bounded by thecorners 1926, 1928, 1936 and 1938 includes less than 50 pixels, then thecandidate object may be rejected.

A pallet object may be defined by first, second and third rectangles1802, 1804 and 1810, respectively, as shown in FIG. 18, wherein thefirst and second rectangles 1802, 1804 may correspond to at leastportions of the first and second fork receiving openings 210 and 212 ofthe pallet P illustrated in FIG. 3 and the third rectangle 1810 maycorresponds to a center stringer 208 of the pallet P. Given theestimated upper left hand corner and the estimated upper right handcorner of the center stringer, a height h, of the rectangles 1802, 1804,and 1810 can be calculated using the following process depicted in FIG.21A. FIG. 21B provides a framework to help understand the geometry usedfor the calculations of the process of FIG. 21A.

The image analysis computer 110, in step 2101, retrieves a number ofvalues that were previously calculated or determined. The term“retrieve” is not meant to convey any particular storage/retrievaldevice, method or protocol but simply that these values are accessibleby the image analysis computer 110 when making additional calculations.Some of the values retrieved are the coordinates for the estimated upperleft corner and the estimated upper right corner, wherein the Y valuefor the upper left corner and the Y value for the upper right cornerwere found by determining the distance from the prequalified lower leftcorner/upper surface 200A of the bottom pallet board to the point on theupper pallet board 202, which distance was then subtracted from each ofthe Y values for the prequalified lower left corner and the possiblelower right corner (the distance is subtracted because y-coordinatevalues decrease as a point moves upwards in the first Ro image). Othervalues include the parameters (ρ_(BPBL), φ_(BPBL)) defining the bottompallet board line L_(BPB) passing through the prequalified lower leftcorner.

In step 2103, the image analysis computer 110 calculates a value haccording to the following equation:

$\begin{matrix}{h = {\max\left\{ \begin{matrix}{{round}\left( {\rho_{BPBL} + {x_{UpperLeft}\sin\;\varphi_{BPBL}} - {y_{UpperLeft}\cos\;\varphi_{BPBL}}} \right)} \\{{round}\left( {\rho_{BPBL} + {x_{UpperRight}\sin\;\varphi_{BPBL}} - {y_{UpperRight}\cos\;\varphi_{BPBL}}} \right)}\end{matrix} \right.}} & {{Equation}\mspace{14mu} 11}\end{matrix}$

ρ_(BPBL)=the orthogonal distance from the origin point on the normalizedgray scale image to the bottom pallet board line L_(BPB) passing throughthe prequalified lower left corner;

φ_(BPBL)=is an angle between a horizontal line and the bottom palletboard line L_(BPB);

y_(UpperLeft)=the Y coordinate value for the estimated upper leftcorner;

y_(UpperRight)=the Y coordinate value for the estimated upper rightcorner;

x_(UpperLeft)=the X coordinate value for the estimated upper leftcorner; and

x_(UpperRight)=the X coordinate value for the estimated upper rightcorner.

Referring to FIG. 21B, the process of FIG. 21A may be more easilyunderstood. A prequalified lower left corner 2106, an estimated upperleft corner 2108, and an estimated upper right corner 2110 are depicted.The bottom pallet board line 2102 calculated earlier and passing throughthe lower left corner 2106, has an orthogonal distance ρ_(BPBL) from theorigin O, calculated from Equation 5 above, and forms an angle φ_(BPBL)2104 relative to a horizontal line. That same angle φ_(BPBL) can be usedto draw a first line 2112 that passes through the estimated upper leftcorner 2108 and a second line (not shown) that passes through theestimated upper right corner 2110. If the estimated upper left corner2108 is appropriately aligned with the estimated upper right corner 2110then the same line may pass through both corners. However, it is morelikely that two different lines could be drawn. Each of these two lineswill have a respective orthogonal distance from the origin O. Equation11, uses these respective orthogonal distances to determine which of thetwo possible lines is closer to the origin O and produces the highervalue for h.

In FIG. 21B, the upper left corner 2108 was chosen, simply by way ofexample, as producing the line 2112 with the smaller orthogonal distancefrom the origin O and thus the higher value for h. Thus the line 2112 isdrawn through the estimated upper left corner 2108 and its orthogonaldistance to the origin can be represented byρ_(TPBL) =−X _(UpperLeft) sin φ_(BPBL) +Y _(UpperLeft) cos φ_(BPBL)

Thus, the h found from Equation 11 is the difference between theorthogonal distance ρ_(BPBL) to the bottom pallet board line 2102 andthe orthogonal distance ρ_(TPBL) to the upper line 2112. In making thedetermination in Equation 11, the image analysis computer 110 can roundeach of the two values to the nearest respective integer beforecomparing them or can compare the raw values and then round just themaximum value to determine h.

As mentioned earlier with regards to FIG. 18, a pallet object may bedefined by first, second and third rectangles 1802, 1804 and 1810,respectively, that may represent at least portions of the first andsecond fork receiving openings and a vertical, center stringer of apallet.

The first rectangle 1802 can be defined such that it extends at most 200pixels to the left of the prequalified lower left corner 1806 along thebottom pallet board line L_(BPB) 1805 and extends orthogonally to theline 1805 over y by h pixels. As noted above, the first rectangle 1802potentially represents a portion of the first fork receiving opening 210to the left of a pallet center stringer. Also, the second rectangle 1804can be defined such that it extends at most 200 pixels to the right ofthe possible lower right corner 1808, which possible lower right corner1808 comprises a candidate object with the prequalified lower leftcorner 1806, along a line 1825 and extends orthogonally to the line 1825over y by h pixels. The line 1825 may or may not pass through theprequalified lower left corner 1806; however, it is oriented withrespect to the x-axis by the same angle φ_(BPBL). The second rectangle1804 potentially represents a portion of the second fork receivingopening 212 to the right of the center stringer (corresponding to thethird rectangle 1810). The third rectangle 1810 extends horizontallyfrom the prequalified lower left corner 1806 to the candidate object'spossible lower right corner 1808 and extends orthogonally, over y, by hpixels to potentially represent a pallet center stringer.

The width of the first rectangle 1802 and the second rectangle 1804 canvary depending on the size of the image, the x-coordinate value of theprequalified lower left corner X_(LLC) and the x-coordinate value of thepossible lower right corner, X_(LRC). For an image that is 480×752pixels in size, the respective maximum width of the first and secondrectangles 1802, 1804 may be 200 pixels. For an image that is 240×376pixels in size, the respective maximum width of the first and secondrectangles 1802, 1804 may be 100 pixels. However, the candidate objectmay have a prequalified lower left corner and possible lower rightcorner located in the image such that an edge of the image area iswithin 200 pixels (or 100 pixels) of one of the corners. In thisinstance, the width of a rectangle (e.g., 1802, 1804) may have to beless than 200 pixels (or 100 pixels if the camera does 2×2 binning)Thus, as explained in more detail below, a leftWidth value is calculatedand a rightWidth value is calculated that determine the respective widthof the first rectangle 1802 and the second rectangle 1804.

For example, if the prequalified lower left corner 1806 is located at aposition (145, 250), the possible lower right corner 1808 is located ata position (155, 252), the width of each rectangle is a respectivecalculated value, h=10, and the angle φ_(BPBL)=8°, then all of thecorners 1801, 1821, 1803, 1815, 1806, 1808, 1809, 1811, 1823, 1813,1817, and 1807 of each of the three triangles 1802, 1804, and 1810 canbe calculated. As described more fully below, the values of the pixelsfrom the normalized gray scale image that form the different rectangles1802, 1804, and 1810 may then be used to calculate a value thatrepresents the confidence that these rectangles actually correspond topallet holes and a center stringer.

While the precise bounds of the rectangles 1802, 1804, and 1810 can becalculated and used in producing such a confidence value, someassumptions and approximations can alternatively be made that simplifysome of the computational steps described below. In particular each ofthe rectangles 1802, 1804, and 1810 can be approximated by a respectiveparallelogram, wherein each parallelogram is constructed from h lines ofpixels wherein each line of pixels is parallel to the bottom palletboard line 1805. Thus, as the term is used herein, “rectangle” is alsointended to encompass a “parallelogram” as well. As explained earlier, aquick line drawing algorithm, such as Bresenham's algorithm, can be usedto identify intermediate pixels for inclusion on a line when the twoendpoints of the line are known.

FIG. 23A provides a flowchart of one example process that may be used toconstruct the respective parallelograms that approximate the rectangles1802, 1804 and 1810. FIG. 22A illustrates a parallelogram correspondingto the first rectangle 1802. In general terms, the parallelogram isconstructed by a) determining the pixels of a first line drawn betweenthe two lower corners, b) identifying the two endpoints of a next line,c) identifying the pixels for inclusion in this next line, and d)repeating these steps for h lines. Ultimately, all the pixels of aparticular parallelogram will have been identified and can logically begrouped into h different lines, if desired.

As shown in FIG. 22A, using the lower corners 1801 and 1821 of the firstrectangle 1802, the calculated value h, and the angle φ_(BPBL)/aparallelogram 2202 can be constructed that approximates the firstrectangle 1802 utilizing the process of FIG. 23A. The process of FIG.23A may be used with any of the three parallelograms but the examplediscussion that follows happens to involve the first parallelogram 2202corresponding to the first rectangle 1802.

At step 2302, the x-coordinate, x₁₁, of the left endpoint (i.e., thelower left corner 1801 for the first line of the parallelogram 2202) iscalculated according to:x ₁₁=round(X _(LLC)−leftWidth*cos(φ_(BPBL))), see FIG. 18,

and the x-coordinate, x₁₂, of the right endpoint (i.e., the lower rightcorner 1821 for the first line of the parallelogram 2202) is calculatedaccording to:x ₁₂ =X _(LLC)−1.

In the equations above the first number in the subscript refers to therectangle corresponding to the parallelogram and the second number inthe subscript refers to the endpoint for a particular line in theparallelogram. For example, “x₁₂” refers to the x-coordinate of thesecond (i.e., right) endpoint of a current line for the parallelogram2202 corresponding to the first rectangle 1802. However, as more fullydescribed below with respect to the y-coordinates of the respectiveendpoints, each parallelogram includes h different lines. Therefore,another index value, r, can be added in the subscript to distinguishwhich of the particular one of the h lines is being utilized in anequation. This index value, r, ranges in value from r=0 to r=h−1 withr=0 referring to the bottom-most line of a parallelogram. Thus, aninitial step 2301 may be performed which initializes the value of r tobe equal to zero. Using this subscript convention, see FIG. 18, thex-coordinates of the endpoints for the first line in each of the threeparallelograms can be calculated according to:x _(11r)=round(X _(LLC)−leftWidth*cos(φ_(BPBL)))x _(12r) =X _(LLC)−1x _(21r) =X _(LRC)+1x _(22r)=round(X _(LRC)+rightWidth*cos(φ_(BPBL)))x _(31r) =X _(LLC)x _(32r) =X _(LRC)

where for the first line, r=0; and

${leftWidth} = {\min\left( {{{floor}\left( \frac{X_{LLC}}{\cos\;\varphi_{BPBL}} \right)},{maxWidth}} \right)}$${rightWidth} = {\min\left( {{{floor}\left( \frac{{imageWidth} - X_{LRC}}{\cos\;\varphi_{BPBL}} \right)},{maxWidth}} \right)}$${maxWidth} = \left\{ \begin{matrix}{100,{{for}\mspace{14mu} 240 \times 376\mspace{14mu}{image}\mspace{14mu}{size}}} \\{200,{{for}\mspace{14mu} 480 \times 752\mspace{14mu}{image}\mspace{14mu}{size}}}\end{matrix} \right.$

and imageWidth is the number of columns of pixels in the image.

In an example above, the prequalified lower left corner 1806 is locatedat a position (145, 250), the possible lower right corner 1808 islocated at a position (155, 252) and the angle φ_(BPBL)=8°. If the imagesize is assumed to be 240×376 pixels, then according to the aboveequations leftWidth=100 and rightWidth=100. In an alternative examplewith the image size being 480×752 pixels, the above equations providethat leftWidth=146 and rightWidth=200. According to the abovecalculations, the lower right corner 1821 of the parallelogram 2202 isat the same x-location, x₁₂, as the lower right corner 1821 of the firstrectangle 1802 and is selected to be one pixel to the left of theprequalified lower left corner 1806. The lower left corner 1801 of theparallelogram 2202 is at the same x-location, x₁₁, as the lower leftcorner 1801 of the rectangle 1802 and is located leftWidth pixels, alongthe bottom pallet board line 1805, to the left of the prequalified lowerleft corner 1806.

Next, the image analysis computer 110 can calculate the y-coordinatesfor these two endpoints, respectively, y₁₁ and y₁₂. For example, they-coordinate for the lower left corner 1801 can be calculated from theequation for the orthogonal distance ρ_(line) to a line that passesthrough a known point (x₁₁, y₁₁) at a known angle φ_(BPBL):ρ_(line) =−x ₁₁ sin φ_(BPBL) +y ₁₁ cos φ_(BPBL)

solving for the y-coordinate of the point gives the equation:y ₁₁=(ρ_(line) +x ₁₁ sin φ_(BPBL))/cos φ_(BPBL)

However, as mentioned, the process of FIG. 23A automates the calculationof the endpoints for h different lines. Therefore, in step 2306, theimage analysis computer 110 uses the integer, index value “r” whichvaries from 0 to h−1 to reference each individual line in theparallelogram 2202. FIG. 23B illustrates 3 of the h−1 lines that arecalculated for the first parallelogram 2202. The bottom most line is thebottom pallet board line 1805 and corresponds to r=0. The next line up,line 2301, corresponds to r=1 and has an orthogonal distance from theorigin equal to (ρ_(BPBL)−r). There are h of these lines calculated withthe last line, line 2303, corresponding to r=h−1 and having anorthogonal distance from the origin equal to (ρ_(BPBL)−h+1).

Thus, for any of the lines (e.g., r=0 to r=h−1) the y-coordinate of theleft endpoint (i.e., y_(11r)) and the y-coordinate of the right endpoint(i.e., y_(12r)) can be calculated, by the image analysis computer 110 instep 2308, according to:y _(11r)=round((ρ_(BPBL) −r+x _(11r)*sin φ_(BPBL))/cos φ_(BPBL)))y _(12r)=round((ρ_(BPBL) −r+x _(12r)*sin φ_(BPBL))/cos φ_(BPBL))).With the two coordinates of the two endpoints for the current line beingknown, the image analysis computer 110 can, in step 2310, identify theintermediate pixel locations to include in the current line extendingbetween the two endpoints using a conventional line drawing algorithmsuch as Bresenham's algorithm. The pixel locations identified forinclusion in a line are then used to extract the pixel values at thoselocations in the normalized gray scale image. As described below, thesepixel values are used to generate a respective vertical projection foreach of the three parallelograms that correspond to the three rectangles1802, 1804, and 1810.

In step 2312, the image analysis computer 110 determines if all h lineshave been calculated or if more remain. If more remain, the newx-coordinates are calculated in steps 2302 and 2304 and the r indexvalue is incremented in step 2306. In setting the new x-coordinatevalues in steps 2302 and 2304, an approximation is made that the sidesof the parallelogram are orthogonal in the y direction. Thus, x_(11r)and x_(12r) are the same for each of the h lines and steps 2303 and 2304need only calculate these values the first time the algorithm of FIG.23A executes.

To summarize, the respective first and second endpoint coordinates foreach of the h lines of the first parallelogram 2202 are calculatedaccording to:x _(11r)=round(X _(LLC)−leftWidth*cos(φ_(BPBL)));x _(12r) =X _(LLC)−1;y _(11r)=round((ρ_(BPBL) −r+x _(11r)*sin φ_(BPBL))/cos φ_(BPBL))), andy _(12r)=round((ρ_(BPBL) −r+x _(12r)*sin φ_(BPBL))/cos φ_(BPBL)))

wherein:

-   -   X_(LLC)=an x-coordinate for the prequalified lower left corner        through which the bottom pallet board line, L_(BPB), passes, and    -   r=an index value uniquely referring to one of the respective h        lines in each of the first parallelogram 2202, the second        parallelogram 2220 and the third parallelogram 2240, the value        of which ranges from 0 to (h−1) with r=0 referring to a        bottom-most of the h lines.

Also, the respective first and second endpoint coordinates for each ofthe h lines in the second parallelogram 2220 are calculated accordingto:x _(21r) =X _(LRC)+1;x _(22r)=round(X _(LRC)+rightWidth*cos(φ_(BPBL)));y _(21r)=round((ρ_(BPBL) −r+x _(21r)*sin φ_(BPBL))/cos φ_(BPBL))), andy _(22r)=round((ρ_(BPBL) −r+x _(22r)*sin φ_(BPBL))/cos φ_(BPBL)))

wherein:

-   -   X_(LRC)=an x-coordinate for a possible lower right corner of a        possible center stringer, and    -   r=an index value uniquely referring to one of the respective h        lines in each of the first parallelogram 2202, the second        parallelogram 2220 and the third parallelogram 2240, the value        of which ranges from 0 to (h−1) with r=0 referring to a        bottom-most of the h lines.

Finally, for the third parallelogram 2240, the respective first andsecond endpoint coordinates for each of the h lines are calculatedaccording to:x _(31r) =X _(LLC);x _(32r) =X _(LRC);y _(31r)=round((ρ_(BPBL) −r+x _(31r)*sin φ_(BPBL))/cos φ_(BPBL))), andy _(32r)=round((ρ_(BPBL) −r+x _(32r)*sin φ_(BPBL))/cos φ_(BPBL))).

The process of FIG. 23A highlighted how the h lines were calculated.However, the reason the h lines are calculated relates to the verticalprojection that was calculated in step 2310. The details of this stepare illustrated in the flowchart of FIG. 23C. The process of FIG. 23Cbegins when two endpoints for a particular line of the parallelogram2202 have been calculated. Based on these two endpoints, a fast linedrawing algorithm can be used to determine which intermediate pixels areincluded in the particular line.

In step 2350, the image analysis computer 110, begins at the leftendpoint on its first iteration (i.e., i=1). As described in more detailabove, a fast line drawing algorithm, such as Bresenham's algorithm, insubsequent iterations, steps in the x-direction by one pixel eachiteration using the slope of the line to determine the y coordinatevalue for that particular iteration of the algorithm. Thus, in step2352, the image analysis computer 110 determines the x-coordinate andthe y-coordinate for a current pixel location at the i^(th) iteration ofthe algorithm for a particular line r. That value of the pixel in thenormalized gray scale image at that pixel location is determined in step2354.

In the example parallelogram 2202, each line of the parallelogram (i.e.,each different value of “r”) spans leftCount pixels in the x-direction,where leftCount=round(leftWidth*cos(φ_(BPBL))), and, thus, there areleftCount iterations performed to determine the intermediate pixels ofeach line. A respective accumulator Y_(i) is used to construct avertical projection of negative pixels for each of the leftCountdifferent iterations of the algorithm 2310. Hence, in the illustratedembodiment, leftCount accumulators are defined corresponding toleftCount iterations performed for each of the h lines of theparallelogram. Each accumulator also corresponds to a respectiveX-coordinate value for one of the leftCount pixels in each of the hlines.

In step 2356, it is determined if the pixel value of the normalized grayscale image at the pixel location just generated during the i^(th)iteration is negative. If so, then that pixel value is added to theaccumulator Y_(i) in step 2358. The value of a pixel in the normalizedgray scale image is negative when not over pallet structure and tends tobe positive when over pallet structure. The image analysis computer 110,in step 2360, increments the index count, i, and the x-coordinate value.In step 2362, the image analysis computer 110 determines if allleftCount iterations have been completed. If not, then the processrepeats starting at step 2352 with the next intermediate pixel locationin the line.

At the end of the steps of the process of FIG. 23C, the intermediatepixel values for a particular line have been calculated and used toupdate a respective accumulator Y_(i) for each iteration of thefast-line drawing algorithm. Because the over-arching process of FIG.23A is repeated for the h lines, this process of FIG. 23C is repeatedfor each of the h lines and, therefore, the accumulators Y_(i) representdata for all the lines that are used to construct the parallelogram2202. Only negative pixels are accumulated into projections.

Another way to describe how the value in an accumulator Y_(i) iscalculated is to consider the process described above as accumulating ineach respective accumulator Y_(i) a sum according to:

$Y_{i} = {\sum\limits_{r = 0}^{r = {h - 1}}{GSPV}_{ir}}$

where:

-   -   i=an index value referring a relative x-coordinate value of a        particular pixel location in a particular line of the first        parallelogram 2202, ranging in value from 1 to Q, where i=1        corresponds to the left-most pixel location in the particular        line and i=Q corresponds to the right-most pixel location in the        particular line;    -   r=an index value uniquely referring to one of the respective h        lines in each of the first parallelogram 2202, the second        parallelogram 2220 and the third parallelogram 2240, the value        of which ranges from 0 to (h−1) with r=0 referring to a        bottom-most of the h lines; and    -   GSPV_(ir)=a gray scale pixel value (GSPV) of a pixel from the        normalized gray scale image at a pixel location indicated by an        i^(th) position in the r^(th) line of the first parallelogram        2202, wherein GSPV_(ir) is added to the respective accumulator        sum Y_(i) if GSPV_(ir) is less than zero.

Referring back to FIG. 22A, the corners of example parallelogram 2202have respective coordinate values of:

-   -   Lower left: (45,236)    -   Lower right: (144,250)    -   Upper left: (45,227)    -   Upper right: (144,240)

Each line of the parallelogram (i.e., each different value of “r”) spans100 pixels in the x-direction, in this particular example, and arespective accumulator Y_(i) is used to construct a vertical projectionfor each of the 100 different columns of the parallelogram 2202. Forexample, for the bottom line 1805, the fast line drawing algorithm stepsfrom x=45 to x=144 and calculates an appropriate y-coordinate value eachstep of the way. In the example of FIG. 22A, the 96^(th) step of thealgorithm is highlighted for each line. For the line r=0, the 96^(th)step has a pixel location (140,247); for the line r=4, the 96^(th) stephas a pixel location (140, 243); and for the line r=9, the 96^(th) stephas a pixel location (140,238).

If the parallelogram 2202 is overlaid onto the normalized gray scaleimage, then the accumulator Y₉₆ can be seen as the vertical projectionof the negative pixels from column x=140 of the normalized gray scaleimage within the y bounds of the parallelogram 2202 for that column.

As mentioned, the processes of FIG. 23A and FIG. 23B were described withrespect to the parallelogram 2202 that corresponds to the firstrectangle 1802. However, a parallelogram 2220 as shown in FIG. 22B cansimilarly be constructed that corresponds to the second rectangle 1804.Also, a parallelogram 2240 as shown in FIG. 22C can be similarlyconstructed that corresponds to the third rectangle 1810.

The parallelogram 2220 of FIG. 22B is substantially similar to firstparallelogram 2202 in that there are rightCount accumulators 2222 thatare used for determining the vertical projection of the pixels from theparallelogram 2220. More generally though, each line of theparallelogram 2220 spans rightCount pixels in the x-direction whererightCount=round(rightWidth*cos(φ_(BPBL))), thereby dictating that thereare rightCount accumulators 2222. However, in other examples, the valuesof leftCount and rightCount may be different so a different number ofaccumulators may be used. The parallelogram 2240 of FIG. 22C correspondsto the third rectangle 1810 which might represent a center stringer.Thus, the width of this parallelogram is not determined by rightWidth orleftWidth values but is determined by the coordinates of theprequalified lower left corner 1806 and the possible lower right corner1808.

For purposes of generating a score for a “hole,” the first and secondparallelograms 2202 and 2220 are treated as one unit. Thus, one verticalprojection for the first and second parallelograms 2202 and 2220 iscalculated and a separate, second vertical projection for the thirdparallelogram 2240 is calculated.

As mentioned above, the first and second parallelograms 2202 and 2220are treated as a single unit when calculating the vertical projectionfor these regions. Thus, according to the processes of FIG. 23A and FIG.23C, 200 values of Y_(i) are calculated, one for each column of the twoparallelograms 2202 and 2220. Much like the process for calculating theaccumulator values Y_(i) for the first parallelogram 2202, the values ofthe accumulators Y_(i) for the second parallelogram 2220 can also becalculated. For the second parallelogram 2220, the value in anaccumulator Y_(i) of negative pixels is calculated by accumulating ineach respective accumulator Y_(i) a sum according to:

$Y_{i} = {\sum\limits_{r = 0}^{r = {h - 1}}{GSPV}_{ir}}$

where:

-   -   i=an index value referring a relative x-coordinate value of a        particular pixel location in a particular line of the second        parallelogram 2220, ranging in value from Q+1 to Q+rightWidth,        where i=Q+1 corresponds to the left-most pixel location in the        particular line and i=Q+rightWidth corresponds to the right-most        pixel location in the particular line;    -   r=an index value uniquely referring to one of the respective h        lines in each of the first parallelogram 2202, the second        parallelogram 2220 and the third parallelogram 2240, the value        of which ranges from 0 to (h−1) with r=0 referring to a        bottom-most of the h lines; and    -   GSPV_(ir)=a gray scale pixel value (GSPV) of a pixel from the        normalized gray scale image at a pixel location indicated by an        i^(th) position in the r^(th) line of the second parallelogram        2220, wherein GSPV_(ir) is added to the respective accumulator        sum Y_(i) if GSPV_(ir) is less than zero.

Based on the different values for Y_(i) for the first and secondparallelograms 2202 and 2220 that are calculated, an average value canbe determined according to Equation 12:

$\overset{\_}{Y} = {\frac{1}{N}{\sum\limits_{i = 1}^{N}Y_{i}}}$where N represents the total number of columns in the combined twoparallelograms 2202 and 2220 (e.g., using the previous examplecoordinates N=200). This average value can be used when determining ascore for the third parallelogram 2240.

Referring to FIG. 22C, the shaded pixels represent the thirdparallelogram 2240. There is an index value j 2270 that varies from 1 to11 (in this example) to uniquely identify each column of theparallelogram 2240. For each column, a value Z_(j) 2272 is calculated,according to the process of FIG. 23D, that represents the sum of aportion of the negative pixel values in each column of the gray scaleimage.

The process of FIG. 23D is similar to that of FIG. 23C. As describedabove, the process of FIG. 23C is applied to the first and secondparallelograms 2202 and 2202. The process of FIG. 23D is applied in asimilar manner to the third parallelogram 2240. The process begins whentwo endpoints for a particular line of the parallelogram 2240 have beencalculated (e.g., endpoints 1806 and 1808 of the first line of the thirdparallelogram 2240). Based on these two endpoints, a fast line drawingalgorithm can be used to determine which intermediate pixels areincluded in the particular line.

In step 2351, the image analysis computer 110, begins at the leftendpoint (i.e., j=1) on the current line (e.g., line r) of theparallelogram 2240. As described in more detail above, a fast linedrawing algorithm, such as Bresenham's algorithm, in subsequentiterations, steps in the x-direction by one pixel each iteration usingthe slope of the line to determine the y coordinate value for thatparticular iteration of the process of FIG. 23D. Thus, in step 2353, theimage analysis computer 110 determines the x-coordinate and they-coordinate for a current pixel location at the j^(th) iteration of thealgorithm for a particular line r. That value of the pixel in thenormalized gray scale image at that pixel location is determined in step2355.

In the example parallelogram 2240, each line of the parallelogram (i.e.,each different value of “r”) spans from the x-coordinate value X_(LLC)for the prequalified lower left corner 1806 to the x-coordinate valueX_(LRC) for the possible lower right corner 1808 and, thus, there are(X_(LRC)−X_(LLC)) iterations performed to determine the intermediatepixels of each line. A respective accumulator Z_(j) is used to constructa vertical projection of some of the negative pixels for each of thedifferent iterations of the process of FIG. 23D (e.g., in the example ofFIG. 22C, there are 11 iterations). Hence, in the illustratedembodiment, 11 accumulators are defined corresponding to 11 iterationsperformed for each of the h lines of the parallelogram 2240. Eachaccumulator also corresponds to a respective X-coordinate value for oneof the 11 pixels in each of the h lines.

Even if the pixel value for a current pixel location (in step 2355) isnegative, it is not necessarily included in the vertical projectionbeing calculated in a particular accumulator Z_(j). Only those pixelsthat have a gray scale value that satisfies the inequality are included:

${{pixel}\mspace{14mu}{value}} \leq \left( {0.75 \times \frac{\overset{\_}{Y}}{h}} \right)$

wherein Y was calculated according to Equation 12 and h was calculatedaccording to Equation 11.

Thus, in step 2357, it is determined if the pixel value of thenormalized gray scale image at the pixel location just generated duringthe j^(th) iteration satisfies the above inequality. If so, then thatpixel value is added to the accumulator Z_(j), in step 2359. The imageanalysis computer 110, in step 2361, increments the index count, j, andthe x-coordinate value. In step 2363, the image analysis computer 110determines if all iterations have been completed. If not, then theprocess repeats starting at step 2353 with the next intermediate pixellocation in the current line.

At the end of the steps of the process of FIG. 23D, the intermediatepixel values for a particular line have been calculated and used toupdate a respective accumulator Z_(j) for each iteration of thefast-line drawing algorithm. This process of FIG. 23D is repeated foreach of the h lines so that eventually the accumulators Z_(j) representdata for all the lines that are used to construct the parallelogram2240.

Taken together, the values of Z_(j) are the vertical projection of aportion of the negative-valued pixels of the normalized gray scale imagecorresponding to the pixel locations of the parallelogram 2240. Anotherway to describe how the value in an accumulator Z_(j) is calculated isto consider the process described above accumulating in each respectiveaccumulator Z_(j) a sum according to:

$Z_{j} = {\sum\limits_{r = 0}^{r = {h - 1}}{GSPV}_{jr}}$

where:

-   -   j=an index value referring a relative x-coordinate value of a        particular pixel location in a particular line of the third        parallelogram 2240, ranging in value from 1 to s, where j=1        corresponds to the left-most pixel location in the particular        line and j=s corresponds to the right-most pixel location in the        particular line;    -   r=an index value uniquely referring to one of the respective h        lines in each of the first parallelogram 2202, the second        parallelogram 2220 and the third parallelogram 2240, the value        of which ranges from 0 to (h−1) with r=0 referring to a        bottom-most of the h lines; and    -   GSPV_(jr)=a gray scale pixel value (GSPV) of a pixel from the        normalized gray scale image at a pixel location indicated by a        j^(th) position in the r^(th) line of the third rectangle 2240,        wherein GSPV_(jr) is added to the respective accumulator sum        Z_(j) if GSPV_(jr) is less than or equal to

$\left( {0.75 \times \frac{\overset{\_}{Y}}{h}} \right).$

A composite hole score is then calculated, for the three parallelogramsusing Equation 13:

${Score}_{hole} = {1 - {\frac{1}{N}{\sum\limits_{i = 1}^{N}\frac{{Y_{i} - \overset{\_}{Y}}}{{Y_{i} + \overset{\_}{Y}}}}} - \frac{\overset{\_}{Z}}{\left( {\overset{\_}{Z} + \overset{\_}{Y}} \right)}}$where:$\overset{\_}{Y} = {{\frac{1}{N}{\sum\limits_{i = 1}^{N}{Y_{i}\mspace{14mu}{and}\mspace{14mu}\overset{\_}{Z}}}} = {\frac{1}{s}{\sum\limits_{j = 1}^{s}Z_{j}}}}$

In the above calculation, s represents the number of columns from theprequalified lower left corner to the potential lower right corner and Nrepresents the total number of columns in the combined twoparallelograms 2202 and 2220. Using the previous example coordinatess=11 and N=200.

If the respective average of either Y or Z is zero then the compositehole score is set to zero; otherwise a composite hole score iscalculated according to the above equation. The middle term of the righthand side of Equation 13 is at a minimum for rectangular holes and theratio of the mean of the Z projection to the sum of the Z mean and the Ymean (the right-most term in Equation 13) is at a minimum for thevertical stringer. Thus, according to the above equation, the value forScore_(hole) is higher when the first and second parallelograms 2202 and2220 are located over pallet holes and the third parallelogram 2240 islocated over a pallet stringer.

For each candidate object that has a Score_(left) (and/or Score_(right),depending on the particular techniques used to calculate these values)that surpasses a predetermined threshold as described above, a compositeobject score is determined using the following equation:

${Score}_{Object} = {\left( {{Score}_{LowerLeftCorner} + {Score}_{LowerRightCorner}} \right)*{Score}_{BaseboardLine}*\frac{{Score}_{hole}}{2}}$

where:

-   -   Score_(Lower left corner)=The calculated value from the        lower-left corner image for the pixel location corresponding to        the prequalified lower left corner of the candidate object;    -   Score_(Lower right corner)=The calculated value from the        lower-right corner image for the pixel location corresponding to        the possible lower right corner of the candidate object; and

${Score}_{BaseboardLine} = {{\max\left( \frac{{AccumulatorCount}(\varphi)}{M \times N_{y}} \right)} = {{The}\mspace{14mu}{calculated}\mspace{14mu}{score}}}$according to the highest counter value of the 61 lines that wereevaluated with Equation 6. The value “M” is the width, in pixels, of theRo image and N_(y) is the number of sub-lines associated with eachcounter; andScore_(Hole)=The value from Equation 13.

The above description for identifying candidate objects in a gray scaleimage that may correspond to a pallet related to a pallet that can becharacterized as having one center stringer (e.g., 208) separating twoholes or openings of a pallet P. One of ordinary skill will recognizethat similar techniques may be used even if a different pallet structureis present. For example, some pallets may include two center stringersseparating two holes or openings and having a central hole or openingbetween the two center stringers. In this case, each center stringerwill have its own four corners that can be identified using thetechniques described above. Also, the Y and Z values of Equation 13 maybe modified to account for the additional center rectangular hole andthe additional center stringer. Thus, an example composite scorecalculation may be different such as, for example:

${Score}_{Object} = {\left( {{Score}_{LowerLeftCorners} + {Score}_{LowerRightCorners}} \right)*{Score}_{BaseboardLine}*\frac{{Score}_{hole}}{4}}$

Some pallets may have a center stringer with a portion that extendsforward of a bottom pallet board line. The general techniques describedabove may still be used but compensation in the way the 61 lines ofEquation 6 are scored may need to be made. Some pallets may have roundedcorners, or other types of corners, occurring between a bottom palletboard line and a center stringer rather than the square cornersdiscussed earlier with respect to pallet P of FIG. 3. Again the generaltechniques described above may be used, but a different mask structurethan those of masks 300 and 310 (See FIG. 6 and FIG. 7) may providebeneficial results. There may be some pallets that do not include alower pallet board. In this instance, a top of a rack shelf on which thepallet sits can be an alternative structure used to identify a bottompallet board line and the same, or similar, techniques described abovemay be used to identify and score various candidate objects.

The candidate objects within an image for which a composite object scorevalue “Score_(object)” has been determined are referred to herein as“scored candidate objects” and define objects which are most likely tocorrespond to actual pallets in the image. Scored candidate objectstypically have a Score_(object) value between 0 about 1.0 while scoredcandidate objects which actually correspond to a pallet typically have ascore greater than 0.5, approaching 1.0 for either image size. However,there may be a relatively large number of such scored candidate objectsidentified in an image frame even though there are actually only a fewpallets likely in an image frame. Accordingly, the number of scoredcandidate objects within each image frame can be further reduced tolessen the computational burden of subsequent analysis of the pluralityof image frames.

As described in more detail below, the image analysis computer 110acquires a series of image frames at different instances of time. Ineach image frame, candidate objects are identified and scored using theabove-described techniques. One or more of these scored candidateobjects are then tracked, if possible, between the image frames. Toaccomplish this tracking, the image analysis computer 110 maintainsinformation about the existing scored candidate objects from theprevious or prior image frames and the new scored candidate objectsidentified in the most recently acquired or next image frame. In themost recently acquired image frame the “new” scored candidate objectsthat are identified may include some scored candidate objects that havenot previously been identified in any previous image frame but may alsoinclude scored candidate objects that may match, or correspond to, anexisting scored candidate object previously identified in an earlierimage frame.

One way for the image analysis computer 110 to maintain this informationis to maintain a list of all scored candidate objects such that for eachscored candidate object a respective record is stored that contains anumber of corresponding object features. The object features caninclude, for example, the parameters (ρ_(BPBL), φ_(BPBL)) which are theorthogonal distance from the origin and an angle from horizontal for acorresponding bottom pallet board line, a corresponding composite objectscore value “Score_(object)”, x-y coordinates for a lower left corner ofa corresponding center stringer, x-y coordinates for the other threecorners of the center stringer, x-y coordinates for the rectangles 1802,1804, 1810, see FIG. 18, associated with the scored candidate objectthat possibly correspond to left and right fork receiving openings andthe center stringer respectively, the rectangles' height h, ascalculated according to Equation 11 above, and corresponding individualcomponent score values such as Score_(LowerRightCorner),Score_(LowerLeftCorner), and Score_(Holes). As described in more detailbelow, there are other object features that can be included in eachrecord which help track a scored candidate object between differentimage frames.

This list of scored candidate objects, and its corresponding records,can be referred to as an Objectlist. The image analysis computer 110 canmaintain two different Objectlists. An ExistingObjectlist can bemaintained that includes existing scored candidate objects and aNewObjectlist can be maintained that includes the scored candidateobjects in the most recently acquired image frame. FIG. 23E depicts aflowchart of an exemplary process for maintaining Objectlists inaccordance with the principles of the present invention. In step 2370,the image analysis computer 110 maintains an ExistingObjectlist which,as mentioned above, includes a respective record for existing scoredcandidate objects, from one or more of the previous image frames, whichare being tracked. The image analysis computer 110, in step 2372,acquires the next image frame or the next gray scale image.Additionally, the image analysis computer 110 creates a NewObjectlistfor the eventual scored candidate objects in this next image frame.Using the techniques described above, the image analysis computeridentifies and then scores the scored candidate objects in this nextimage frame, in step 2374. In a step more fully described below, theimage analysis computer 110 can prune, in step 2376, scored candidateobjects from the NewObjectlist to reduce the number of scored candidateobjects that will be tracked. The scored candidate objects in the prunedNewObjectlist can be matched, or attempted to be matched, withcorresponding objects in the ExistingObjectlist, in step 2378. Somescored candidate objects in the ExistingObjectlist may have no matchesin the NewObjectlist and therefore may possibly be deleted, in step2380, from the ExistingObjectlist so that they are no longer tracked. Itmay be beneficial to avoid deleting a scored candidate object the firsttime it is unmatched and to wait for either two or three sequentialunmatched occurrences before deleting the scored candidate object. Also,some scored candidate objects in the NewObjectlist may have nocorresponding matching objects in the ExistingObjectlist; these scoredcandidate objects are then added to the ExistingObjectlist.

Depending on how the scored candidate objects match, the records in theExistingObjectlist can be updated to reflect new information from theNewObjectlist, in step 2382, to provide a current list of scoredcandidate objects that will continue to be tracked into the next imageframe. Also, in step 2384, the image analysis computer 110 can add anynewly identified objects in the NewObjectlist that do not find a matchwith an existing object in the ExistingObjectlist to the objects thatare in the ExistingObjectlist. Control of the process then returns tostep 2372 where a next image frame can be acquired and a newNewObjectlist can be created so the process can repeat.

As mentioned above with respect to step 2376 of FIG. 23E, the imageanalysis computer 110 can prune some of the scored candidate objectsfrom the NewObjectlist if desired. In particular, using the objectfeatures associated with each scored candidate object, the imageanalysis computer 110 can eliminate some scored candidate objects fromthe NewObjectlist for an image frame that are unlikely to actuallycorrespond to a pallet.

The steps shown in FIG. 23E are shown as occurring sequentially;however, one of ordinary skill will recognize that some of the steps canbe performed concurrently without departing from the scope of thepresent invention. For example, it is not necessary to wait until allobjects are matched in step 2378 to start the acquisition of a nextimage frame in step 2372.

FIG. 24A depicts a flowchart for an exemplary algorithm to eliminate atleast some of the scored candidate objects from the NewObjectlist for animage frame. As mentioned above, and as shown in step 2402, the imageanalysis computer 110 maintains a NewObjectlist for the scored candidateobjects in the current image frame being analyzed. Using the respectivecomposite object score Score_(object) for each scored candidate object,the image analysis computer 110 can eliminate, in step 2404, all scoredcandidate objects that have a Score_(object) value below a predeterminedminimum value. For example, all scored candidate objects with aScore_(object) score below 0.3 can be deleted from the NewObjectlist forthe image frame.

In accordance with one example aspect of the present invention, agraphical display of the gray scale image can be provided to an operatorof the vehicle 20 via an image monitor (not shown) on the vehicle 20,wherein the graphical display can be controlled by the vehicle computer50. In this graphical display, information representing the scoredcandidate objects in the ExistingObjectlist can be overlayed onto thegray scale image. For example, a red line representing the bottom palletboard line of a scored candidate object can be visibly displayed overthe area of the gray scale image where that bottom pallet board line isbelieved to be located. Similarly, information representing a centerstringer for a scored candidate object can also be overlayed onto thegray scale image within the graphical display. For example, two verticallines, each representing a respective left vertical edge and respectiveright vertical edge of the stringer can be displayed. Alternatively, arespective circle can be displayed that corresponds to a prequalifiedlower left corner and a possible lower right corner of a scoredcandidate object's center stringer. The graphically overlayedinformation can also be color coded so that different scored candidateobjects can be easily distinguished within the graphical display. Toreduce the potential of visual clutter on the display, the imageanalysis computer 110 may maintain scored candidate objects in theExistingObjectlist and the NewObjectlist that have a composite objectscore greater than about 0.3 but only scored candidate objects in theExistingObjectlist having a composite object score greater than or equalto about 0.5 are displayed by the vehicle computer 50.

In addition to eliminating some of the scored candidate objects in step2404, the image analysis computer 110 can also determine if there arescored candidate objects that are too close to other scored candidateobjects. In other words, two pallets cannot physically occupy the samespace which means that if two scored candidate objects are closer to oneanother than the physical dimensions of a pallet would allow, then atleast one of those scored candidate objects should be eliminated. Thus,in step 2406, the image analysis computer identifies and eliminatesscored candidate objects that have Score_(object) values lower than theother scored candidate objects that are nearby. For example, the imageanalysis computer 110 may first determine the location of each scoredcandidate object in the NewObjectlist and then identify those scoredcandidate objects which are within a predetermined distance of oneanother. Scored candidate objects within this predetermined distance ofone another are considered to be nearby one another. Of those scoredcandidate objects which are identified as being nearby one another, someof these scored candidate objects can be eliminated from theNewObjectlist for that image frame. FIG. 24B provides additional detailsof a process that can be implemented to accomplish step 2406.

Additional scored candidate objects can also be eliminated from theNewObjectlist by relying on the dimensions of a respective centerstringer for each scored candidate object. Even though two scoredcandidate objects may be far enough from one another that they are notconsidered to be nearby one another, their coexistence may still beunlikely based on the vertical coordinates of their respective centerstringers. Thus, a subset of scored candidate objects may be consideredvertically local, or just “local,” to one another if portions of theirrespective center stringers overlap a particular vertical span ofcoordinates regardless of their distance in the horizontal direction. Ofthose scored candidate objects considered local to one another, thescored candidate object having the maximum composite scoreScore_(object) can be retained in the NewObjectlist and the other scoredcandidate objects deleted. For example, given a particular scoredcandidate object (e.g., Object(m)) having a bottom pallet board line(ρ_(BPBLm), φ_(BPBLm)), and the maximum value Score_(Object) as comparedto other local candidate objects, where all the other local candidateobjects are located within a predetermined number pixels of the bottompallet board line, the other local candidate objects can be eliminatedfrom the NewObjectlist, in step 2408. FIG. 24C provides additionaldetails of a process that can be implemented to accomplish step 2408.

Thus, using the pruning steps 2404, 2406, and 2408, the NewObjectlistfor an image frame can be reduced in size so that fewer scored candidateobjects are included in the NewObjectlist during subsequent imageprocessing and analysis as described herein.

As mentioned, FIG. 24B provides additional details of a process that canbe implemented to accomplish step 2406 in which scored candidate objectsin the NewObjectlist are further analyzed to determine if one or more ofthe scored candidate objects can be eliminated from the NewObjectlistfor the image frame. Initially, in step 2410, the image analysiscomputer 110 determines if there are any scored candidate objects withinthe NewObjectlist that have not been tested. On the initial pass of theprocess 2406, the entire NewObjectlist remains untested and with eachiteration another scored candidate object is tested until eventually allthe scored candidate objects are tested. For example, the NewObjectlistmay include a plurality of scored candidate objects (e.g., N) and eachobject can be uniquely referenced by an index value k that ranges invalue from 1 to N. In other words, Object(k) can be used to uniquelydenote one of the scored candidate objects included in theNewObjectlist. So, if there are any scored candidate objects that remainto be tested, the control passes on to step 2412; otherwise, controlpasses on to step 2422.

In step 2412, the image analysis computer 110 selects one of thecandidate objects, Object(k), from the NewObjectlist. Onestraightforward way to select an object is to use the index value kwhich is incremented, in step 2420, during each iteration of the process2406 such that for each step the particular scored candidate objectbeing tested can be referred to as Object(k). In step 2414, the imageanalysis computer 110 identifies all the scored candidate objects thatare closer than a predetermined distance to Object(k).

To accomplish the identification of nearby objects, the location of eachscored candidate object may first be determined. As mentioned above,there are a number of object features associated with each of the scoredcandidate objects, including a variety of different x-y coordinate pairsthat refer to different features (e.g., the center stringer lower leftcorner, center stringer lower right corner, etc.) of the candidateobject. The location of a scored candidate object, Object(k), will be aset of x-y coordinates (x_(object(k)),y_(object(k))). One beneficialpoint of reference that can be considered to correspond to the locationof the scored candidate object is the lower center of the centerstringer of that scored candidate object. Referring to FIG. 24D, anObject(k) 2448 is illustrated by depicting a center stringer 2454. Thecenter stringer 2454 includes a prequalified lower left corner 2450 anda possible lower right corner 2452. A bottom pallet board line 2456drawn through the prequalified lower left corner 2450 has an orthogonaldistance from the origin 2400 of ρ_(k) 2460 in the first Ro image. Thecoordinates for location 2458, the object's Lower Center Point, of thescored candidate object 2448 (and the center stringer 2454) can beestimated from the object features related to the scored candidateobject. For example, using the x coordinate of the lower left cornerX_(LLC) 2450 and the x coordinate of the lower right corner X_(LRC) 2452the x coordinate value x_(object(k)) can be estimated to be equal to

$\frac{X_{LLC} + X_{LRC}}{2}.$Using the orthogonal distance ρ_(k) 2460 from the first Ro image for thescored candidate object 2448, the coordinate y_(object(k)) can beestimated to be equal to ρ_(k) 2460. Thus, for each object within theNewObjectlist, a corresponding respective location can be determined.This “location” value may be one of the object features that are storedfor each scored candidate object in the NewObjectlist for an imageframe. Thus, when a particular scored candidate object is identified inthe NewObjectlist, this object feature, or attribute, can be retrievedand used during image analysis and processing of the image frame. One ofordinary skill will recognize that the “location” of a scored candidateobject can be defined by a feature of the scored candidate object otherthan the lower center of the center stringer, without departing from thescope of the present invention.

Once a respective location for each of the scored candidate objects isidentified, then step 2414 can identify all those scored candidateobjects which are closer than a predetermined distance to a particularscored candidate object, Object(k). In a two dimension image frame, suchas the one depicted in FIG. 24D, the distance, d, between two points(x_(g), y_(g)) and (x_(h), y_(h)) can be calculated according tod=√{square root over ((x_(g)−x_(h))²+(y_(g)−y_(h))²)}{square root over((x_(g)−x_(h))²+(y_(g)−y_(h))²)}. Using this calculation for distance,the image analysis computer, in step 2414, determines all scoredcandidate objects in the NewObjectlist that are nearby Object(k) asthose scored candidate objects that satisfy the following equation:

$d \geq \sqrt{\left( {x_{{object}{(k)}} - x_{{object}{(i)}}} \right)^{2} + \left( {y_{{object}{(k)}} - y_{{object}{(i)}}} \right)^{2}}$

where:

k: is an index value of a particular scored candidate object, Object(k),within the NewObjectlist for which the respective distance fromObject(k) to all other scored candidate objects is to be determined; kcan range from 1 to N;

(x_(object(k)),y_(object(k))) are the x-y coordinate pair for the scoredcandidate object, Object(k);

i: is an index value of a particular scored candidate object, Object(i),in the NewObjectlist; other than the value k, the index value i rangesfrom 1 to N so that the distance of all other scored candidate objectsfrom Object(k) can be determined;

(x_(object(i)),y_(object(i))) are the x-y coordinate pair for the scoredcandidate object, Object(i); and

d: is a predetermined allowable minimum distance between scoredcandidate objects, wherein two candidate objects having a distance fromone another that is less than or equal to d are considered to be nearbyone another. The value d, for example, can be 50 pixels.

As a result of testing the above equation, for a particular scoredcandidate object Object(k), against all the other scored candidateobjects within the NewObjectlist, the image analysis computer 110 candetermine if there are any scored candidate objects that are determinedto be nearby Object(k).

Because the coordinate values for a scored candidate object refers topixel locations, the distance between scored candidate objects refers toa distance between pixel locations. However, the distance between pixellocations corresponds to a physical distance between pallet structure,or potential pallet structure, that is being captured in the gray scaleimage. Accounting for the typical sizes of pallets, or a known range ofsizes, and the corresponding scale of the gray scale image, a physicaldistance between pallet structure can be correlated to pixel distance ofthe gray scale image. If two (or more) scored candidate objects arecloser in pixel distance, e.g., 50 pixels, when using a large image size(e.g., 480×752 pixels) or 25 pixels for a smaller image size (e.g.,240×376 pixels), than could be physically possible with two differentpallets, then at least one of those candidate objects can be eliminatedas a possible candidate object because both (or all) of them can notcorrespond to actual pallet structure in the gray scale image.

Thus, in step 2416, the image analysis computer 110 compares therespective Score_(object) for Object(k) to the respective Score_(object)value for all the nearby objects identified in step 2414. If the scoreof Object(k) is less than the score of any of the objects identified instep 2414, then Object(k) is placed on the list for future deletion, instep 2418.

In step 2420, the image analysis computer 110 increments k and repeatsthe testing process for the next scored candidate object of theNewObjectlist. Eventually, all the scored candidate objects in theNewObjectlist are tested and all the scored candidate objects added tothe list for deletion can then be pruned from the NewObjectlist, in step2422. Assuming that one or more scored candidate objects were added tothe deletion list, in step 2418, the index value k for the NewObjectlistnow ranges from 1 to M where M<N. Referring back to FIG. 24D, a circle2446 is shown that circumscribes a region that is d pixels in radius andcentered on the location 2458 of Object(k), i.e., scored candidateobject 2448. Of the remaining scored candidate objects (2466, 2470,2472, 2474, 2486) depicted in FIG. 24D, Object(4) 2486 is considered tobe nearby Object(k) because its location (x₄, ρ₄) 2488 is within thecircle 2446. Assuming that Object(4)2486 has a higher Score_(object)than Object(k) 2448, then Object(k) is added to the deletion list instep 2418.

Under the assumption that the true pallet has the greatest pallet score,it is not possible for the gray scale image to include two (or more)different pallet stringers that overlap within a relatively small bandof y-pixel values. Thus, if two (or more) scored candidate objects haverespective pallet stringers that do overlap in this small band ofy-pixel values of the gray scale image, then all but one of these scoredcandidate objects can be eliminated. As described earlier, the y-pixelvalue for the lower center 2458 of the stringer 2454 can be estimated bydrawing a substantially horizontal line 2456 (e.g., ±15°) through aprequalified lower left corner 2450 of a scored candidate object 2448and calculating an orthogonal distance from the origin 2400 to thatsubstantially horizontal line 2456 using the first Ro image. Thus, bycomparing the respective orthogonal distances calculated for eachrespective center stringer of each scored candidate object, those scoredcandidate objects that have stringers within the same range of y-pixelvalues can be identified as being vertically local, or simply “local,”to one another.

As noted above, FIG. 24D provides an example image frame having aplurality of scored candidate objects 2448, 2466, 2470, 2472, 2474, 2486each represented by a corresponding, respective stringer. Of all thescored candidate objects that are in the NewObjectlist, the strongest,or highest, valued candidate object, i.e., the one having the highestScore_(object) value, for example, Object(m) 2466, can be selected tocalculate the range of y-coordinate values that define the band ofinterest. Object(m) includes object feature ρ_(m) 2471, an orthogonaldistance from an origin to a bottom pallet board line of Object(m).Using this value, the upper y-pixel value 2462 of the band of interestis estimated to be at ρ_(m)−V_(s) while the lower y-pixel value 2464 ofthe band is estimated to be at ρ_(m)+V_(s), where V_(s) represents apredetermined number of pixels referred to as the vertical span and can,for example, be about 50 pixels. All the scored candidate objects thathappen to also have their respective center stringer located within thisband can be identified for elimination because they have lowerScore_(object) values than that of Object(m).

Thus, within that band of y-pixel values, one scored candidate object2466 having the maximum Score_(object) of all the scored candidateobjects within that band is selected to remain in the NewObjectlistwhile all other scored candidate objects in that band of y-pixel valuesare eliminated from the NewObjectlist.

As mentioned earlier, FIG. 24C provides additional details of a processthat can be implemented to accomplish process 2408 that can eliminatescored candidate objects based on their y-coordinate values relative toone another. The process 2408 of FIG. 24C begins with the NewObjectlistof scored candidate objects, which may have been reduced following step2406 in FIG. 24A, and produces a new, potentially smaller list of scoredcandidate objects referred to as KeepObjectlist. The use of twodifferent lists of scored candidate objects is merely one example of away to test and prune scored candidate objects from a list of scoredcandidate objects; one of ordinary skill will recognize that otherfunctionally equivalent methods of maintaining list structures within aprogrammable computer can be utilized without departing from the scopeof the present invention. Furthermore, the use of the term“KeepObjectlist” is used merely for clarity in the following descriptionto distinguish the list of scored candidate objects in the NewObjectlistat the end of the process 2408 as compared to the list of scoredcandidate objects in the NewObjectlist at the beginning of the process2408. At the end of the process 2408, in step 2440, the maintaining of aKeepObjectList refers to the further pruned NewObjectList after all thescored candidate objects identified for inclusion are identified. Thus,for a given image frame, there may be only one list of the scoredcandidate objects in that frame that is maintained by the image analysiscomputer 110 and that list continues to be referred to as theNewObjectlist.

In step 2430, the process is initialized with the KeepObjectlist beingan empty set of scored candidate objects; as the process 2408 executes,this list will be populated with those scored candidate objects thatwill continue to be identified as scored candidate objects in this imageframe. Next, in step 2432, the image analysis computer 110, determinesif the list NewObjectlist is an empty set. Initially, the NewObjectlistcontains all the remaining scored candidate objects from the image framefollowing step 2406 in FIG. 24A and so control passes to step 2434 toidentify a first scored candidate object from the NewObjectlist. Inparticular, in step 2434, the image analysis computer 110 identifies thecandidate object, Object(m), which has the maximum object compositescore value Score_(Object) of all the candidate objects within theNewObjectlist. Also, see FIG. 24D, this candidate object 2466 hasassociated object feature ρ_(m) 2471, an orthogonal distance from anorigin to a bottom pallet board line of Object(m). This particularscored candidate object is added to the list KeepObjectlist. Next, instep 2436, the image analysis computer 110 identifies all scoredcandidate objects in NewObjectlist that have a location whosey-coordinate location is within a particular span of values.

Similar to the process described with respect to FIG. 24B, the locationof a scored candidate object can be defined a variety of different ways,but one way to define the location of a scored candidate object is toidentify the x-y coordinates of the lower center of the center stringeras the object's location. For any particular scored candidate object,Object(i), those location coordinates can referred to as the pair(x_(LCi), y_(LCi)). Using these respective coordinates for each of thescored candidate objects in the NewObjectlist, the image analysiscomputer 110, in step 2436, can determine which scored candidate objectsin the NewObjectlist are located such that they have a y-coordinatevalue within ±V_(s) of the value ρ_(m) 2471; V_(s) can, for example,equal 50 pixels. More particularly, Object(m) has a bottom pallet boardline 2468 with an orthogonal distance to the origin 2400 of ρ_(m) 2471and an associated angle φ_(m) 2476 from a horizontal line. Each of theother identified scored candidate objects, Object(i), has a respectivelocation (x_(LCi), y_(LCi)) through which a line can be drawn at thesame angle as φ_(m) 2476 to calculate whether a scored candidate objectis within the vertical range of interest with respect to Object(m).

In particular, Object(1) 2470, in FIG. 24D, can be used as an example.The object 2470 has a location 2481 of (x₁, ρ₁) which can be referred togenerically as (x_(LCi), y_(LCi)). A line 2483 can be drawn at an anglethe same as angle φ_(m) so as to pass through the location 2481. Anorthogonal line drawn from the origin 2400 to the line 2483 will also becollinear with the orthogonal line ρ_(m) 2471. The orthogonal distancefrom the origin 2400 to the line 2483 can be calculated and is referredto as ρ_(calci) 2463. In particular, a similar orthogonal distance canbe calculated for any Object(i) in the Objectlist according to theformula:ρ_(calci) =x _(LCi) sin φ_(m) +y _(LCi) cos φ_(m).

Thus, the image analysis computer 110 can use the location (x_(LCi),y_(LCi)) of each object, Object(i), in the NewObjectlist to calculate arespective orthogonal distance, ρ_(calci), from that object's locationto the origin 2400 in order to identify all the other scored candidateobjects that have a respective orthogonal distance within ±V_(s) of thebottom pallet board line 2468 of the Object(m). In other words, allscored candidate objects with stringers satisfying the followinginequality are identified in step 2436:ρ_(m) −V _(s) ≦−x _(LCi) sin φ_(m) +y _(LCi) cos φ_(m)≦ρ_(m) +V_(s)  Equation 14:

All the scored candidate objects satisfying the above inequality, aswell as Object(m), are then removed, in step 2438, from theNewObjectlist by the image analysis computer 110. As noted above,Object(m), i.e., scored candidate object 2466, was previously added tothe list KeepObjectlist.

In the example image frame of FIG. 24D, only Object(2) 2472 satisfiesthe inequality of Equation 14 when Object(m) comprises scored candidateobject 2466. When the process 2408 of FIG. 24C executes, Object(2) 2472is eliminated from the NewObjectlist in step 2438.

These steps are repeated until the image analysis computer 110determines, in step 2432, that the NewObjectlist is empty. In step 2440,after the NewObjectlist is empty, the KeepObjectlist is defined as anupdated NewObjectlist, which may comprise a pruned version of theprevious NewObjectlist with which the process 2408 started. This versionof the NewObjectlist is used in analyzing the current version of theExistingObjectlist, as described in detail below. As a result of suchanalysis, the ExistingObjectlist is updated based on the information inthe NewObjectlist. Once the ExistingObjectlist is updated, a next imageframe may be available for which a next NewObjectlist is created. Thus,a NewObjectlist is conceptually the same for each image frame but theactual information and values in the NewObjectlist will differ for eachacquired image frame. FIG. 25 depicts a flowchart of an exemplaryprocess for tracking the scored candidate objects through different grayscale image frames to further refine the determination of where a palletstructure is located within the image frames. As the fork carriageapparatus 40 of the vehicle 10 moves, the location of the scoredcandidate objects within an image captured by an imaging camera 130 onthe fork carriage apparatus 40 also appears to move within the differentimage frames. This information from a series of captured image framesassist in determining which scored candidate objects correspond toactual pallet structure.

In FIG. 25, step 2502 represents that the image analysis computer 110maintains an ExistingObjectlist, as discussed above, that includesrecords from one or more previous image frames for the existing scoredcandidate objects that are presently being tracked. Scored candidateobjects are assigned to separate threads of tracked objects with eachthread having a unique identification tag, Tag_(Object). As mentioned,one attribute of each of the scored candidate objects in theExistingObjectlist is that object's respective location. As before, thelocation of a scored candidate object can refer to the coordinates ofthe lower center of the center stringer for that scored candidateobject. For example, an object, Object(r), 2602, shown in FIG. 26A, hasa location that is referred to by x-y coordinates (x_(r), y_(r)). Thesescored candidate objects 2602, 2604 of FIG. 26A are scored candidateobjects that have been identified and scored according to the techniquesdescribed above. Each of the scored candidate objects are maintained inthe ExistingObjectlist.

Each thread of tracked objects operates on a prediction and update cyclewith each thread having its own predictor. As described below in detail,using a Kalman filter to accomplish such tracking is one approach;however, one of ordinary skill will recognize that there are other knowntechniques to accomplish such tracking, as well. Given a change in forkheight a prediction is made of where the object moved to in the nextimage frame, observations are made of actual object locations, if theappropriate conditions are met, these new observations are matched toexisting objects, and finally the object location predictor is eitherupdated using the new observations or the object thread is terminatedand deleted.

As described earlier, an imaging camera 130 can be coupled with the forkcarriage apparatus 40 of a vehicle 10. Because the movement of the forkcarriage apparatus 40 occurs in a controlled manner, the resultingdisplacement of the imaging camera 130 that occurs between two momentsin time can be calculated. For example, knowing the velocity that thefork carriage apparatus 40 is moving allows the camera displacementbetween image frames to be easily calculated. Alternatively, an encoderand wire or cable assembly coupled between the fork carriage assembly 40and the third weldment 36 may be provided for generating pulses to thevehicle computer 50 in response to movement of the fork carriageapparatus 40 relative to the third weldment 36 so as to allow thevehicle computer 50 to determine the linear displacement of the imagingcamera 130. Because two successive image frames captured by the imagingcamera 130 will have been taken from different perspectives, the scoredcandidate objects will appear to be in different locations in the twodifferent image frames. These two successive image frames can be moreeasily referred to as “a prior image frame” and “a next image frame.”Based on an amount that the height of the fork carriage apparatus 40 ofa vehicle 10 changes between a prior image frame and a next image frame,in step 2503, the image analysis computer 110 can respectively predict,in step 2504, where each of the scored candidate objects from the priorimage frame should be located in the next image frame that is capturedfrom the new position of the imaging camera 130. The image analysiscomputer 110 will also define a prediction window surrounding eachpredicted location in the next image frame for each scored candidateobject from the prior image frame. The prediction window(s) may be avariety of different sizes but embodiments herein contemplate predictioncircles with a radius of about 50 pixels for a larger image size and 25pixels for a smaller image size. One of ordinary skill will recognizethat other sizes and shapes of prediction windows may be selectedwithout departing from the scope of the present invention.

For example, referring to FIG. 26A, there are three candidate objects,Object(r) 2602, Object(s) 2604 and Object(t) 2606, in theExistingObjectlist for the previous image frame. These objects can bereferred to as “existing objects” or “existing scored candidateobjects.” They have respective locations (x_(r), y_(r)), (x_(s), y_(s))and (x_(t), y_(t)) that represent their location at the time the priorimage frame was taken or acquired. When a next image frame is captured,as shown in FIG. 26B, with the camera placement for the next image framebeing a known distance from the camera placement for the prior imageframe, then the respective locations (x′_(r), y′_(r)) 2602′, (x′_(s),y′_(s)) 2604′ and (x′_(t), y′_(t)) 2606′ can be predicted in the nextimage frame of FIG. 26B. Further in FIG. 26B, prediction window 2601defines a circular region centered around the predicted location 2602′of Object(r), prediction window 2603 defines a circular region centeredaround the predicted location 2604′ of Object(s) and prediction window2605 defines a circular region centered around the predicted location2606′ of Object(t).

The image analysis computer 110 acquires, in step 2505, the next imageframe at the new camera location and then, in step 2506, determines thescored candidate objects in that next image frame and their respectivelocations. Thus, concurrent with acquiring the next image frame in step2505, the image analysis computer 110 may also acquire, in step 2503,the amount that the height of the fork carriage apparatus 40 of thevehicle 10 changes between the prior image frame and the next imageframe. As noted above, objects in a next image frame can be referred toas “new objects” or “new scored candidate objects.” As shown in FIG.26B, the NewObjectlist for the next image frame includes five newobjects, Object(a) 2610, Object(b) 2612, Object(c) 2614, Object(d) 2616and Object(e) 2618. These scored candidate objects are considered to bea new set of scored candidate objects from the next image frame. Usingthe newly determined locations of the new set of scored candidateobjects, the image analysis computer 110 further determines, in step2510, if there are any new scored candidate objects that are located ineach prediction window 2601, 2603, 2605.

The image analysis computer may identify a prediction window in the nextimage frame that does contain one or more scored candidate objects. Inthis case the image analysis computer 110, in step 2510, tentativelyidentifies that the existing scored candidate object from the priorimage frame has a matching new scored candidate object in the next imageframe. If only one new scored candidate object is within the predictionwindow in the next image frame, then that new scored candidate object isconsidered tentatively matched with the existing scored candidate objectfrom the prior image frame (that was used to calculate the predictionwindow). If, however, two or more new scored candidate objects arewithin a prediction window in the next image frame, then furtheranalysis is performed to select only one of these scored candidateobjects. As the same new object could conceivably match to more than oneexisting object all matches are considered tentative and are resolved instep 2512.

The prediction window 2601 of FIG. 26B is an example of two or more newscored candidate objects being located within a prediction window. Inparticular, the NewObjectlist for the next image frame includesObject(a) 2610 and Object(b) 2612, which are both within the predictionwindow 2601 that corresponds to the predicted location of Object(r) fromthe first image frame. Each of these candidate objects has a respectivecomposite object score Score_(Object) that is used, in step 2512, by theimage analysis computer to determine whether Object(a) or Object(b) ismatched to Object(r). Each existing object is not considered to be trulymatched until all matches are resolved by step 2512.

In particular, the absolute values:|Score_(Object)(r)−Score_(Object)(a)| and|Score_(Object)(r)−Score_(Object)(b)| are evaluated such that the newobject having the smallest absolute value (i.e., the new object that hasthe closest Score_(Object) value compared to Object(r)) is consideredthe tentatively matching new scored candidate object within theprediction window 2601. The absolute score difference is minimized overall tentatively matched objects for a particular existing object. Inother words, for a particular existing scored candidate object, the newscored candidate object in its tentative match list will be chosen asthe existing scored candidate object's final match that minimizes theabsolute score difference.

For example, in the image frame of FIG. 26B, the new scored candidateobject Object(b) may be tentatively matched with both Object(r) andObject(s) because it is located in both prediction circles 2601 and2603. Thus in step 2512, the image analysis computer 110 also evaluatesthe absolute values: |Score_(Object)(s)−Score_(Object)(b)| and|Score_(Object)(r)−Score_(Object)(b)|, with the smallest absolute valuerevealing which of the existing objects is matched with the new objectObject(b). When a new object is matched with an existing object, thenthe record for that existing object in the ExistingObjectlist willinclude updated information such as the new object's new location,updated prediction factors, and its MatchCount value (the predictionfactors and MatchCount value are described in more detail below).

If a respective prediction window for a scored candidate object from theexisting objects does not include any new scored candidate objects, thenthe existing scored candidate object is likely a falsely identifiedscored candidate object and may be discarded. However, because oflighting, shadows or other artifacts introduced during acquisition of agray scale image, an actual pallet object may not be discernible in aparticular image frame even though it is discernible in other imageframes. Thus, a single instance of an existing object not being matchedwith a new object may not result in the existing object being identifiedas an object to be discarded. A value, MatchCount, can be introduced asone of the attributes that are stored for an existing object in theExistingObjectlist.

As mentioned, the current version of the ExistingObjectlist includesscored candidate objects. At least some of these scored candidateobjects were also in the previous version of the ExistingObjectlist.That previous version of the ExistingObjectlist had its owncorresponding NewObjectlist with which it was analyzed. Based on thatprior analysis, the previous version of the ExistingObjectlist wasupdated based upon the information about scored candidate objects fromits corresponding NewObjectlist. The updating of the previous version ofthe ExistingObjectlist resulted in the current version of theExistingObjectlist and this current version of the ExistingObjectlist isnow being analyzed with respect to its own corresponding NewObjectlistof scored candidate objects in the next image frame.

During this present analysis, the respective value for MatchCount for anexisting object in the current version of the ExistingObjectlistprovides an indication of whether that object failed to have a matchingnew object in the old NewObjectlist that was compared with the previousversion of the ExistingObjectlist. For example, during the currentanalysis, if MatchCount presently equals “1”, then that indicates thatthis existing scored candidate object failed to have a match in theprevious analysis when the previous version of the ExistingObjectlistwas analyzed with respect to its own corresponding NewObjectlist.

When an existing object is ultimately resolved to match a new object, instep 2512, the MatchCount for that existing object in theExistingObjectlist is set to a predetermined value, such as “2”, by theimage analysis computer 110 in step 2514.

In step 2516, the image analysis computer 110, identifies existingscored candidate objects, if any, that were not matched with one of thenew objects. The image analysis computer 110, in step 2518, thenevaluates the respective MatchCount value for that existing object todetermine if it is equal to “0”. If so, then that existing object is nolonger considered a scored candidate object to be tracked in subsequentimage frames and, in step 2520, it is discarded. If the MatchCount valuefor the unmatched existing object is not equal to “0”, then the existingobject is not discarded but, in step 2522, its MatchCount value isdecremented. In FIG. 26B, there is no new object located within theprediction window 2605 for Object(t). Thus, in this example, existingscored candidate Object(t) would not have a matching new object in theNewObjectlist.

As will be described in more detail below, the past locations of thescored candidate objects and the accuracy with which a prediction windowwas calculated and matched can be useful in refining the way in whichsubsequent prediction windows are calculated. Thus, in step 2524, theimage analysis computer 110 updates the factors that it uses tocalculate a respective prediction window in a subsequent image frame foreach of the scored candidate objects.

The image analysis computer 110, in step 2526, identifies new scoredcandidate objects in the next image frame that do not correspond to anyexisting scored candidate objects from the previous image frames. Forexample, in FIG. 26B, there is a new scored candidate object Object(c)2614 that is located outside of the prediction windows 2601, 2603, 2605and 2607. Thus, this is a new scored candidate object that has nocorresponding existing scored candidate object in theExistingObjectlist. In step 2526, the image analysis computer 110 addsthis new candidate object to the ExistingObjectlist and sets itsMatchCount value equal to “2”.

As mentioned above, Object(c) 2614 is a new object with no matchingexisting object so Object(c) is added to the updated ExistingObjectlist.There is also another possibility of a new object with no matchingexisting object. In FIG. 26A, there is an existing Object(u) 2608. InFIG. 26B there is a predicted location 2608′ and a prediction window2607. Within the prediction window 2607 are two new scored candidateobjects, Object(d) 2616 and Object(e) 2618. As mentioned above, in step2512, the respective composite object scores of Object(u), Object(e) andObject(d) are used to resolve the matching pair of objects. If forexample, Object(e) 2618 is determined to be the match for Object(u)2608, then Object(d) 2616 does not match any existing object in theExistingObjectlist. Thus, Object(d) 2616 is added to the updatedExistingObjectlist as a new scored candidate object to start tracking.

Using the updated ExistingObjectlist, the image analysis computer 110 instep 2528, can provide updated object information to the vehiclecomputer 50 so that the graphical display of the existing scoredcandidate objects can be updated.

The process of FIG. 25 then repeats itself for a third image frame, afourth image frame, etc. by returning to step 2504 for each iteration.

The values associated with pruning and predicting scored candidateobjects may be adjusted according to the image size. For example, thevalues provided above may be appropriate for images which are 480×752pixels in size. However, for images which are 240×376 pixels in size,the values may be adjusted so that each is approximately half of theexample values provided above. For example V_(S) can be 25 pixelsinstead of 50 pixels, the prediction window (e.g., 2607 of FIG. 26B) canhave a radius of about 25 pixels instead of 50 pixels, and the value fordefining nearby objects (see step 2406 of FIG. 24A) can be 25 pixelsinstead of 50 pixels.

FIG. 27A depicts a pallet 2702 in the physical world that has a lowercenter point that represents the location 2704 of the pallet 2702. Thelocation 2704 of the pallet 2702 depends on what frame of reference, or3D coordinate system, is used to define the coordinates for the palletlocation 2704. As mentioned earlier, one aspect of the present inventionrelates to detecting information beneficial to placing forks 42A, 42B ofa vehicle 10 at one or more openings of a pallet. Thus, a world 3Dcoordinate system (also referred to herein as “the world coordinatesystem”) used to define the location of the pallet 2702 can have itsorigin 2706 located at a fixed location either on or near the vehicle10, such as a fixed location on the mast assembly having substantiallythe same Y location as the forks 42A, 42B when in their lower, homeposition. In this embodiment, the world coordinate origin 2706 is fixedsuch that the fork carriage apparatus 40 and the forks 42A, 42B moverelative to the fixed world coordinate origin. If the world coordinatesystem is oriented such that the Z direction is parallel with the forks42A, 42B and the Y direction is parallel to the direction of verticalmovement the forks 42A, 42B up and down, then the world coordinateorigin 2706 can be defined such that coordinates (X_(w), Y_(w), Z_(w))of the pallet location 2704 can be determined. In particular, thesecoordinates essentially represent a vector from the world coordinateorigin 2706 to the lower center point (i.e., the location 2704) of thepallet 2702. The world 3D coordinate system could also be located infront of the vehicle 10 on the floor and below the forks 42A, 42B. Insuch an embodiment, the world 3D coordinate system would still be fixedwith respect to the vehicle 10 and move with the vehicle 10. Truckmotion in the X and Z directions may be accounted for via truckodometry.

The scene in the physical world being imaged by the imaging camera 130can also be referred to with reference to a location 2708 of the imagingcamera 130. This is helpful because transformation operations betweenpixel locations in an image and locations of physical objects may besimplified when the locations of the physical objects are considered asbeing relative to the camera location 2708. The camera coordinate systemis also a 3D coordinate system having three mutually orthogonaldirection axes extending from an origin point. Both the orientation ofthe camera coordinate system's directional vectors and the location ofthe camera coordinate system's origin can differ from those of the worldcoordinate system. Conventionally, it is known that transformation fromone 3D coordinate system to another can be accomplished using a rotationmatrix and a translation vector that represent the differences betweenthe two coordinate systems. If (X_(c), Y_(c), Z_(c)) are the coordinatesof the pallet location 2704 in terms of the camera coordinate system,then

$\begin{matrix}{\begin{bmatrix}\begin{matrix}X_{c} \\Y_{c}\end{matrix} \\Z_{c}\end{bmatrix} = {{\left\lbrack R_{cam} \right\rbrack\begin{bmatrix}\begin{matrix}X_{w} \\Y_{w}\end{matrix} \\Z_{w}\end{bmatrix}} + \begin{bmatrix}{tc}_{x} \\{tc}_{y} \\{tc}_{z}\end{bmatrix}}} & {{Equation}\mspace{14mu} 15\; A}\end{matrix}$

The coordinates (X_(w), Y_(w), Z_(w)) and the coordinates (x_(c), y_(c),Z_(c)) both identify the location of the same point 2704 in the physicalworld. One set of coordinates (X_(w), Y_(w), Z_(w)) is with reference tothe world coordinate origin 2706 and the other set of coordinates(x_(c), y_(c), Z_(c)) is with reference to the camera location 2708. InEquation 15A, [R_(cam)] is a 3×3 rotation matrix between the twocoordinate systems' axes and the right-most term is the translationvector, T_(cam)=[tc_(x), tc_(y), tc_(z)]^(T), representing thetranslation between the origin 2706 of the world coordinate systemrelative to the origin 2708 of the camera coordinate system. If theimaging camera 130 is attached to the carriage apparatus 40, then as thecarriage apparatus 40 including the forks 42A and 42B move up and down,the translation, tc_(y), of the imaging camera 130 relative to the worldcoordinate origin 2706 will vary. As shown in FIG. 27A, this translationvalue, tc_(y), can have one component, t_(y), that represents an amountof movement along the mast assembly 30 that the forks 42A and 42B havemoved relative to the fixed world coordinate origin 2706 and can have asecond component, t_(ybias), that represents a fixed vertical offsetbetween the imaging camera 130 relative to the forks 42A and 42B. As theforks 42A and 42B move, the value of t_(y) may vary; but the value oft_(ybias) will remain the same. As mentioned earlier, the imaging camera130 may be fixed to the carriage apparatus such that it is in a positionbelow the forks 42A and 42B (i.e., t_(ybias)<0) but as the imagingcamera moves upward, the Y-coordinate of the camera coordinate systemorigin location 2708 may be located above the world coordinate systemorigin 2706 (i.e., (t_(y)+t_(ybias))>0) as shown in FIG. 27A. Also, asdiscussed below there may be a horizontal offset value t_(xbias) aswell.

FIG. 27A depicts an embodiment in which the camera coordinate system hasits axes rotated about the X-axis 2707 by an angle θ_(e) 2701. Thus, theZ-axis of the camera coordinate system is referred to as Z_(θe) 2703 todistinguish it from the world coordinate origin Z-axis 2705 and theY-axis of the camera coordinate system is referred to as Y_(θe) 2709 todistinguish it from the world coordinate origin Y-axis 2711. Themagnitude of the angle θ_(e) 2701 may, for example, be about 2.2263°;and, from the point of reference of the camera coordinate system, theZ-axis of the world coordinate system is tilted downward at an angle ofabout −2.2263°. In this embodiment, the coordinates (X_(c), Y_(c),Z_(c)) of the pallet location 2704 in Equation 15A would be defined interms of the camera coordinate system axes (X_(θe), Y_(θe), Z_(θe)).

FIG. 27B depicts the geometry of an ideal pinhole camera model. Themodel for an ideal pinhole camera involves a projection of the point(X_(c), Y_(c), Z_(c)) 2704 through the camera location 2708 onto animage plane 2712 that is a distance f, in the Z_(θe) direction, from thecamera location 2708. There is a camera center 2714 which is theprojection of the camera location 2708 in the Z_(θe) direction onto theimage plane 2712. In the illustrated embodiment, distance f is a fixeddistance equal to the camera focal length in the Z_(θe) direction.

Thus, the geometry of FIG. 27B provides the basis for the well-knownpinhole camera projection model for the pixel image coordinates:

$x_{col} = {{f_{x}\frac{X_{c}}{Z_{c}}} + x_{0}}$$y_{row} = {{f_{y}\frac{Y_{c}}{Z_{c}}} + y_{0}}$

where:

-   -   (X_(C), Y_(C), Z_(C)) are the three dimensional coordinates of        the pallet location 2704 in the camera coordinate system;    -   (x_(col), y_(row)) are the pixel location coordinates on the        image plane 2712 of where the pallet location 2704 projects        according to the pinhole camera model;    -   (x₀, y₀) is the camera center 2714 and is the pixel location        coordinates on the image plane 2712 where the camera center (or        the camera coordinate system origin) 2708 projects according to        the pinhole camera model;    -   f_(x) is the focal length, f, expressed in x-direction, or        horizontal, pixel-related units; and    -   f_(y) is the focal length, f, expressed in y-direction, or        vertical, pixel related units.

The above equations can be written in homogenous coordinates accordingto:

$\begin{matrix}{{{\lambda\begin{bmatrix}x_{col} \\y_{row} \\1\end{bmatrix}} = {\begin{bmatrix}f_{x} & 0 & x_{0} \\0 & f_{y} & y_{0} \\0 & 0 & 1\end{bmatrix}\begin{bmatrix}\begin{matrix}X_{c} \\Y_{c}\end{matrix} \\Z_{c}\end{bmatrix}}}{{{where}\mspace{14mu}\lambda} = Z_{c}}} & {{Equation}\mspace{14mu} 16}\end{matrix}$

Equation 15, from earlier, can also be written as:

$\begin{matrix}{\begin{bmatrix}\begin{matrix}X_{c} \\Y_{c}\end{matrix} \\Z_{c}\end{bmatrix} = {\begin{bmatrix}\; & {tc}_{x} \\\left\lbrack R_{cam} \right\rbrack & {tc}_{y} \\\; & {tc}_{z}\end{bmatrix}\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w} \\1\end{bmatrix}}} & {{Equation}\mspace{14mu} 17\; A}\end{matrix}$

Using Equation 17A, the projection transformation of Equation 16 can bewritten as:

$\begin{matrix}{{\lambda\begin{bmatrix}x_{col} \\y_{row} \\1\end{bmatrix}} = {{\begin{bmatrix}f_{x} & 0 & x_{0} \\0 & f_{y} & y_{0} \\0 & 0 & 1\end{bmatrix}\begin{bmatrix}\; & {tc}_{x} \\\left\lbrack R_{cam} \right\rbrack & {tc}_{y} \\\; & {tc}_{z}\end{bmatrix}}\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w} \\1\end{bmatrix}}} & {{Equation}\mspace{14mu} 18}\end{matrix}$

In general terms, Equation 18 can be written as:

$\begin{matrix}{{{\lambda\begin{bmatrix}x_{col} \\y_{row} \\1\end{bmatrix}} = {{\lbrack K\rbrack\left\lbrack R_{cam} \middle| T_{cam} \right\rbrack}\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w} \\1\end{bmatrix}}}{{{where}\mspace{14mu}\lbrack K\rbrack} = \begin{bmatrix}f_{x} & 0 & x_{0} \\0 & f_{y} & y_{0} \\0 & 0 & 1\end{bmatrix}}} & {{Equation}\mspace{14mu} 19\; A}\end{matrix}$

The 3×4 matrix [R_(cam)|T_(cam)] is often referred to as the extrinsiccamera matrix. The matrix [K] can be referred to as the intrinsic cameramatrix for the imaging camera 130. Typically, the values for the matrix[K] may be determined using a camera resectioning process, as is knownin the art. In general, an intrinsic camera matrix has the form:

$\lbrack K\rbrack = \begin{bmatrix}f_{x} & \gamma & x_{0} \\0 & f_{y} & y_{0} \\0 & 0 & 1\end{bmatrix}$

-   -   where:    -   (x₀, y₀) is the camera center 2714 and is the pixel location        coordinates on the image plane 2712 where the camera center (or        the camera coordinate system origin) 2708 projects according to        the pinhole camera model;    -   f_(x) is the focal length, f, expressed in x-direction, or        horizontal, pixel-related units; and    -   f_(y) is the focal length, f, expressed in y-direction, or        vertical, pixel related units, and γ represents a skew factor        between the x and y axes, and is often “0”.

Example values for the intrinsic camera matrix, for a 480×752 pixelimage, can be:

f_(x)=783.1441, f_(y)=784.4520, γ=0, x₀=237.8432, y₀=380.7313

Example values for the intrinsic camera matrix, for a 240×376 pixelimage, can be:

f_(x)=458.333344, f_(y)=458.333344, γ=0, x₀=142.282608, y₀=171.300568

Equation 19A relates the pallet location 2704, up to a scale factor λ,with a gray scale image pixel location (x_(col), y_(row)). This impliesthat the left and right sides of Equation 19A are collinear vectorswhich results in a zero cross-product. The cross product equation is:

$\begin{matrix}{{\begin{bmatrix}x_{col} \\y_{row} \\1\end{bmatrix} \times {{\lbrack K\rbrack\left\lbrack R_{cam} \middle| T_{cam} \right\rbrack}\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w} \\1\end{bmatrix}}} = 0} & {{Equation}\mspace{14mu} 20\; A}\end{matrix}$

Prudent selection of the rotation matrix [R_(cam)] and the translationvector T_(cam) can simplify the calculations involved in Equation 20A.In particular, if the world coordinate system axes and the cameracoordinate system axes are mutually aligned with one another, then therotation matrix is simply a 3×3 identity matrix. However, the camera maytypically be rotated upwards an elevation angle, θ_(e), as shown in FIG.27A, such that relative to the camera Z_(θe)-axis, the world coordinateZ-axis is tilted downward by an angle −θ_(e). Such a difference in theaxes results one possible rotation matrix of:

$\begin{matrix}{\left\lbrack R_{{cam}_{1}} \right\rbrack = \begin{bmatrix}1 & 0 & 0 \\0 & {\cos\;\theta_{e}} & {{- \sin}\;\theta_{e}} \\0 & {\sin\;\theta_{e}} & {\cos\;\theta_{e}}\end{bmatrix}} & {{Equation}\mspace{14mu} 21}\end{matrix}$Additionally, the camera location 2708 can be selected so that it istranslated relative to the world coordinate origin 2706 by a translationvector, as mentioned above, of:

$\begin{bmatrix}{tc}_{x} \\{tc}_{y} \\{tc}_{z}\end{bmatrix}.$In an exemplary embodiment, the imaging camera 130 is attached to thevehicle frame in such a way that it has the same Z-coordinate value asthe world coordinate origin 2706 (i.e., tc_(z)=0) but has anX-coordinate value offset from the world coordinate origin 2706 by abias value (i.e., tc_(x)=t_(xbias)). As mentioned above, as the forkcarriage apparatus 40 moves up and down, the translation of the camera130 relative to the world coordinate origin 2706 also changes in the Ydirection. Accordingly, the camera translation vector may be realizedas:

$\begin{bmatrix}{tc}_{x} \\{tc}_{y} \\{tc}_{z}\end{bmatrix} = \begin{bmatrix}t_{xbias} \\{t_{y} + t_{ybias}} \\0\end{bmatrix}$

Given that:

${\lbrack K\rbrack = \begin{bmatrix}f_{x} & 0 & x_{0} \\0 & f_{y} & y_{0} \\0 & 0 & 1\end{bmatrix}},{\left\lbrack R_{{cam}_{1}} \middle| T_{cam} \right\rbrack = \begin{bmatrix}1 & 0 & 0 & t_{xbias} \\0 & {\cos\;\theta_{e}} & {{- \sin}\;\theta_{e}} & \left( {t_{y} + t_{ybias}} \right) \\0 & {\sin\;\theta_{e}} & {\cos\;\theta_{e}} & 0\end{bmatrix}}$

Equation 20A simplifies to:

$\begin{bmatrix}x_{col} \\y_{row} \\1\end{bmatrix} \times {\quad{{\begin{bmatrix}f_{x} & {x_{0}\sin\;\theta_{e}} & {x_{0}\cos\;\theta_{e}} & {f_{x}t_{xbias}} \\0 & {{f_{y}\cos\;\theta_{e}} + {y_{0}\sin\;\theta_{e}}} & {{{- f_{y}}\sin\;\theta_{e}} + {y_{0}\cos\;\theta_{e}}} & {f_{y}\left( {t_{y} + t_{ybias}} \right)} \\0 & {\sin\;\theta_{e}} & {\cos\;\theta_{e}} & 0\end{bmatrix}\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w} \\1\end{bmatrix}} = 0}}$

which simplifies to Equation 22:

$\begin{bmatrix}x_{col} \\y_{row} \\1\end{bmatrix} \times {\quad{\begin{bmatrix}{{X_{w}f_{x}} + {Y_{w}x_{0}\sin\;\theta_{e}} + {Z_{w}x_{0}\cos\;\theta_{e}} + {f_{x}\left( t_{xbias} \right)}} \\{{Y_{w}\left( {{f_{y}\cos\;\theta_{e}} + {y_{0}\sin\;\theta_{e}}} \right)} + {Z_{w}\left( {{{- f_{y}}\sin\;\theta_{e}} + {y_{0}\cos\;\theta_{e}}} \right)} + {f_{y}\left( {t_{y} + t_{ybias}} \right)}} \\{{Y_{w}\sin\;\theta_{e}} + {Z_{w}\cos\;\theta_{e}}}\end{bmatrix} = 0}}$

calculating the first two elements of the cross product provides twolinear equations referred to collectively as Equation 23:−X _(w) f _(x) −Y _(w) x ₀ sin θ_(e) −Z _(w) x ₀ cos θ_(e) +x _(col)(Y_(w) sin θ_(e) +Z _(w) cos θ_(e))−f _(x)(t _(xbias))=0 y _(row)(Y _(w)sin θ_(e) +Z _(w) cos θ_(e))−Y _(w)(f _(y) cos θ_(e) +y ₀ sin θ_(e))−Z_(w)(−f _(y) sin θ_(e) +y ₀ cos θ_(e))−f _(y)(t _(y) +t _(ybias))=0

In matrix form, the above two equations can be written as Equation 24A:

${\begin{bmatrix}{- f_{x}} & {\left( {x_{col} - x_{0}} \right)\sin\;\theta_{e}} & {\left( {x_{col} - x_{0}} \right)\cos\;\theta_{e}} \\0 & {{\left( {y_{row} - y_{0}} \right)\sin\;\theta_{e}} - {f_{y}\cos\;\theta_{e}}} & {{\left( {y_{row} - y_{0}} \right)\cos\;\theta_{e}} + {f_{y}\sin\;\theta_{e}}}\end{bmatrix}\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w}\end{bmatrix}} = {\quad\begin{bmatrix}{f_{x}\left( t_{xbias} \right)} \\{f_{y}\left( {t_{y} + t_{ybias}} \right)}\end{bmatrix}}$

Thus, for a particular gray scale image which is acquired when theimaging camera 130 is positioned at a location (t_(xbias),(t_(y)+t_(ybias))) relative to the world coordinate origin 2706, thereis an image pixel location (x_(col), y_(row)) which corresponds to thelocation (X_(w), Y_(w), Z_(w)) 2704 of the pallet 2702. Thus, a scoredcandidate object having an image pixel location (x_(col), y_(row)) (See,for example, FIG. 24D object 2448 and location 2458) also has anassociated location (X_(w), Y_(w), Z_(w)) 2704 in the physical world.Thus, this scored candidate object in image space has associated with ita height Y_(w) that is the height in the physical world of the palletthat corresponds to that scored candidate object. The height Y_(w) cansimply be referred to as “the height of the scored candidate object” asa shorthand way of stating that the scored candidate object at location(x_(col), y_(row)) in image space has an associated height Y_(w) in thephysical world of a corresponding pallet at location 2704. As discussedbelow, the image pixel location (x_(col), y_(row)) corresponds to ameasured value in image space. If the imaging camera 130 changes itsposition and a second gray scale image is acquired then Equation 24Astill holds true for (X_(w), Y_(w), Z_(w)) but there will be differentvalues for t_(y) and (x_(col), y_(row)). Thus, for each different imageframe acquired there are respective values for t_(y) and (x_(col),y_(row)), so Equation 24A can more clearly be written as:

$\mspace{641mu}{{{Equation}\mspace{14mu} 25\;{{A\begin{bmatrix}{- f_{x}} & {\left( {x_{{col}_{n}} - x_{0}} \right)\sin\;\theta_{e}} & {\left( {x_{{col}_{n}} - x_{0}} \right)\cos\;\theta_{e}} \\0 & {{\left( {y_{{row}_{n}} - y_{0}} \right)\sin\;\theta_{e}} - {f_{y}\cos\;\theta_{e}}} & {{\left( {y_{{row}_{n}} - y_{0}} \right)\cos\;\theta_{e}} + {f_{y}\sin\;\theta_{e}}}\end{bmatrix}}\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w}\end{bmatrix}}} = {\quad\begin{bmatrix}{f_{x}t_{xbias}} \\{f_{y}\left( {t_{y_{n}} + t_{ybias}} \right)}\end{bmatrix}}}$

where n is an index value denoting the image frame in a sequence ofacquired image frames, t_(xbias) is the horizontal translation of theimaging camera 130 from the world coordinate origin 2706, and t_(y) _(n)+t_(ybias) is the vertical translation of the imaging camera 130 fromthe world coordinate origin 2706 corresponding to when image frame n isacquired. The image pixel location (x_(col) _(n) , y_(row) _(n) ) can bereferred to as one “observation” of the pallet location 2704 in imagespace. The pair of equations in Equation 25A can be collected togetherfor multiple observations of the same pallet location 2704 to provide aseries of equations, as follows:

$\begin{matrix}{\mspace{616mu}{{Equation}\mspace{14mu} 26{A\begin{bmatrix}{- f_{x}} & {\left( {x_{{col}_{1}} - x_{0}} \right)\sin\;\theta_{e}} & {\left( {x_{{col}_{1}} - x_{0}} \right)\cos\;\theta_{e}} \\0 & {{\left( {y_{{row}_{1}} - y_{0}} \right)\sin\;\theta_{e}} - {f_{y}\cos\;\theta_{e}}} & {{\left( {y_{{row}_{1}} - y_{0}} \right)\cos\;\theta_{e}} + {f_{y}\sin\;\theta_{e}}} \\{- f_{x}} & {\left( {x_{{col}_{2}} - x_{0}} \right)\sin\;\theta_{e}} & {\left( {x_{{col}_{2}} - x_{0}} \right)\cos\;\theta_{e}} \\0 & {{\left( {y_{{row}_{2}} - y_{0}} \right)\sin\;\theta_{e}} - {f_{y}\cos\;\theta_{e}}} & {{\left( {y_{{row}_{2}} - y_{0}} \right)\cos\;\theta_{e}} + {f_{y}\sin\;\theta_{e}}} \\{- f_{x}} & {\left( {x_{{col}_{3}} - x_{0}} \right)\sin\;\theta_{e}} & {\left( {x_{{col}_{3}} - x_{0}} \right)\cos\;\theta_{e}} \\0 & {{\left( {y_{{row}_{3}} - y_{0}} \right)\sin\;\theta_{e}} - {f_{y}\cos\;\theta_{e}}} & {{\left( {y_{{row}_{3}} - y_{0}} \right)\cos\;\theta_{e}} + {f_{y}\sin\;\theta_{e}}} \\\vdots & \vdots & \vdots\end{bmatrix}}{\quad{\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w}\end{bmatrix} = \begin{bmatrix}{f_{x}t_{xbias}} \\{f_{y}\left( {t_{y_{1}} + t_{ybias}} \right)} \\{f_{x}t_{xbias}} \\{f_{y}\left( {t_{y_{2}} + t_{ybias}} \right)} \\{f_{x}t_{xbias}} \\{f_{y}\left( {t_{y_{3}} + t_{ybias}} \right)} \\\vdots\end{bmatrix}}}}} & \;\end{matrix}$

This equation can be conceptually written as Equation 27:

${\lbrack A\rbrack\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w}\end{bmatrix}} = \lbrack B\rbrack$which can be manipulated to form:

$\mspace{641mu}{{{Equation}\mspace{14mu} 28\;{A\mspace{79mu}\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w}\end{bmatrix}}} = {{{{\left( {\lbrack A\rbrack^{T}\lbrack A\rbrack} \right)^{- 1}\lbrack A\rbrack}^{T}\lbrack B\rbrack}\mspace{14mu}{where}{\text{:}\lbrack A\rbrack}} = {\quad{{\begin{bmatrix}{- f_{x}} & {\left( {x_{{col}_{1}} - x_{0}} \right)\sin\;\theta_{e}} & {\left( {x_{{col}_{1}} - x_{0}} \right)\cos\;\theta_{e}} \\0 & {{\left( {y_{{row}_{1}} - y_{0}} \right)\sin\;\theta_{e}} - {f_{y}\cos\;\theta_{e}}} & {{\left( {y_{{row}_{1}} - y_{0}} \right)\cos\;\theta_{e}} + {f_{y}\sin\;\theta_{e}}} \\{- f_{x}} & {\left( {x_{{col}_{2}} - x_{0}} \right)\sin\;\theta_{e}} & {\left( {x_{{col}_{2}} - x_{0}} \right)\cos\;\theta_{e}} \\0 & {{\left( {y_{{row}_{2}} - y_{0}} \right)\sin\;\theta_{e}} - {f_{y}\cos\;\theta_{e}}} & {{\left( {y_{{row}_{2}} - y_{0}} \right)\cos\;\theta_{e}} + {f_{y}\sin\;\theta_{e}}} \\{- f_{x}} & {\left( {x_{{col}_{3}} - x_{0}} \right)\sin\;\theta_{e}} & {\left( {x_{{col}_{3}} - x_{0}} \right)\cos\;\theta_{e}} \\0 & {{\left( {y_{{row}_{3}} - y_{0}} \right)\sin\;\theta_{e}} - {f_{y}\cos\;\theta_{e}}} & {{\left( {y_{{row}_{3}} - y_{0}} \right)\cos\;\theta_{e}} + {f_{y}\sin\;\theta_{e}}} \\\vdots & \vdots & \vdots\end{bmatrix}\mspace{20mu}{{and}\mspace{79mu}\lbrack B\rbrack}} = \begin{bmatrix}{f_{x}t_{xbias}} \\{f_{y}\left( {t_{y_{1}} + t_{ybias}} \right)} \\{f_{x}t_{xbias}} \\{f_{y}\left( {t_{y_{2}} + t_{ybias}} \right)} \\{f_{x}t_{xbias}} \\{f_{y}\left( {t_{y_{3}} + t_{ybias}} \right)} \\\vdots\end{bmatrix}}}}}$

Once there are two or more observations for a pallet location 2704,Equation 28A provides more equations than unknowns and can be solvedusing a least-squares method to estimate the pallet location (X_(w),Y_(w), Z_(w)) in terms of the world coordinate origin 2706.Additionally, waiting until there is at least some minimum distancebetween two observations is beneficial as well. For example, it ispreferred that there be at least a minimum fork difference (i.e., |t_(y)_((i+1)) −t_(y) _(i) |≧1 inch), before a pallet location (X_(w), Y_(w),Z_(w)) is estimated.

The rotation matrix [R_(cam) ₁ ] of Equation 21, developed using theconfiguration of FIG. 27A, only contemplates rotation of the cameracoordinate system relative to the X-axis (e.g., angle θ_(e) 2701) of theworld coordinate system. However, the rotation matrix for a camera canbe generalized to account for rotation of the camera coordinate systemrelative to all three axes of the world coordinate system, as shown inFIG. 27E. In FIG. 27E, the axes (2707′, 2711′, and 2705′) of the cameracoordinate system are shown relative to the axes of the world coordinatesystem (2707, 2711, 2705). In particular, θ_(x) 2762 can represent therotation of the camera coordinate system relative to the X-axis 2707(referred to as an elevation angle), θ_(y) 2764 can represent therotation of the camera coordinate system relative to the Y-axis 2711(referred to as a pitch angle), and θ_(z) 2766 can represent therotation of the camera coordinate system relative to the Z-axis 2705(referred to as a deflection angle). As a result a camera coordinatesystem will result having its axes (2707′, 2711′, and 2705′) rotatedwith respect to the world coordinate system axes (2707, 2711 and 2705)by an amount that represents a combination of the individual rotationangles θ_(x) 2762, θ_(y) 2764, and θ_(z) 2766. Each angle of rotationprovides one component of the overall rotation matrix for the camera.

$\begin{matrix}{\left\lbrack R_{{cam}_{x}} \right\rbrack = \begin{bmatrix}1 & 0 & 0 \\0 & {\cos\;\theta_{x}} & {{- \sin}\;\theta_{x}} \\0 & {\sin\;\theta_{x}} & {\cos\;\theta_{x}}\end{bmatrix}} & {{Equation}\mspace{14mu} 29\; A} \\{\left\lbrack R_{{cam}_{y}} \right\rbrack = \begin{bmatrix}{\cos\;\theta_{y}} & 0 & {\sin\;\theta_{y}} \\0 & 1 & 0 \\{{- \sin}\;\theta_{y}} & 0 & {\cos\;\theta_{y}}\end{bmatrix}} & {{Equation}\mspace{14mu} 29\; B} \\{\left\lbrack R_{{cam}_{z}} \right\rbrack = \begin{bmatrix}{\cos\;\theta_{z}} & {{- \sin}\;\theta_{z}} & 0 \\{\sin\;\theta_{z}} & {\cos\;\theta_{z}} & 0 \\0 & 0 & 1\end{bmatrix}} & {{Equation}\mspace{14mu} 29\; C}\end{matrix}$

The more generalized, or full, rotation matrix is the multiplication ofthese three component matrices together to provide Equation 29D:

$\left\lbrack R_{{cam}_{full}} \right\rbrack = {\quad\begin{bmatrix}{\cos\;\theta_{y}\cos\;\theta_{z}} & {{- \sin}\;\theta_{z}\cos\;\theta_{y}} & {\sin\;\theta_{y}} \\{{\sin\;\theta_{x}\sin\;\theta_{y}\cos\;\theta_{z}} + {\cos\;\theta_{x}\sin\;\theta_{z}}} & {{{- \sin}\;\theta_{x}\sin\;\theta_{y}\sin\;\theta_{z}} + {\cos\;\theta_{x}\cos\;\theta_{z}}} & {{- \sin}\;\theta_{x}\cos\;\theta_{y}} \\{{{- \sin}\;\theta_{y}\cos\;\theta_{x}\cos\;\theta_{z}} + {\sin\;\theta_{x}\sin\;\theta_{z}}} & {{\cos\;\theta_{x}\sin\;\theta_{y}\sin\;\theta_{z}} + {\sin\;\theta_{x}\cos\;\theta_{z}}} & {\cos\;\theta_{x}\cos\;\theta_{y}}\end{bmatrix}}$

However, rather than a full rotation matrix, aspects of the presentinvention contemplate a rotation matrix with one rotational degree offreedom about the x-axis and another rotational degree of freedom aboutthe z-axis. As described above, the angle of rotation about the x-axisis typically referred to as the elevation angle θ_(e). The angle ofrotation about the z-axis can be referred to as the deflection angle anddenoted by φ_(defl). The three components of this particular examplerotation matrix are provided by:

$\begin{matrix}{\left\lbrack R_{{cam}_{x}} \right\rbrack = \begin{bmatrix}1 & 0 & 0 \\0 & {\cos\;\theta_{e}} & {{- \sin}\;\theta_{e}} \\0 & {\sin\;\theta_{e}} & {\cos\;\theta_{e}}\end{bmatrix}} & {{Equation}\mspace{14mu} 30\; A} \\{\left\lbrack R_{{cam}_{y}} \right\rbrack = \begin{bmatrix}1 & 0 & 0 \\0 & 1 & 0 \\0 & 0 & 1\end{bmatrix}} & {{Equation}\mspace{14mu} 30\; B} \\{\left\lbrack R_{{cam}_{z}} \right\rbrack = \begin{bmatrix}{\cos\;\varphi_{defl}} & {{- \sin}\;\varphi_{defl}} & 0 \\{\sin\;\varphi_{defl}} & {\cos\;\varphi_{defl}} & 0 \\0 & 0 & 1\end{bmatrix}} & {{Equation}\mspace{14mu} 30\; C}\end{matrix}$

Multiplying these three component matrices together provides anexemplary rotation matrix provided by Equation 30D:

$\left\lbrack R_{{cam}_{2}} \right\rbrack = \begin{bmatrix}{\cos\;\varphi_{defl}} & {{- \sin}\;\varphi_{defl}} & 0 \\{\sin\;\varphi_{defl}\cos\;\theta_{e}} & {\cos\;\varphi_{defl}\cos\;\theta_{e}} & {{- \sin}\;\theta_{e}} \\{\sin\;\varphi_{defl}\sin\;\theta_{e}} & {\cos\;\varphi_{defl}\sin\;\theta_{e}} & {\cos\;\theta_{e}}\end{bmatrix}$

Using this rotation matrix (instead of the one of Equation 21) willproduce a pair of linear equations analogous to those of Equation 24A.In particular, these two linear equations are provided by Equation 24B:

$\begin{bmatrix}{{{- f_{x}}\cos\;\varphi_{defl}} + {\left( {x_{col} - x_{0}} \right)\sin\;\varphi_{defl}\sin\;\theta_{e}}} & {{\left( {x_{col} - x_{0}} \right)\cos\;\varphi_{defl}\sin\;\theta_{e}} + {f_{x}\sin\;\varphi_{defl}}} & {\left( {x_{col} - x_{0}} \right)\cos\;\theta_{e}} \\{\left( {{\left( {y_{row} - y_{0}} \right)\sin\;\theta_{e}} - {f_{y}\cos\;\theta_{e}}} \right)\sin\;\varphi_{defl}} & {\left( {{\left( {y_{row} - y_{0}} \right)\sin\;\theta_{e}} - {f_{y}\cos\;\theta_{e}}} \right)\cos\;\varphi_{defl}} & {{\left( {y_{row} - y_{0}} \right)\cos\;\theta_{e}} + {f_{y}\sin\;\theta_{e}}}\end{bmatrix}{\quad{\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w}\end{bmatrix} = \begin{bmatrix}{f_{x}\left( t_{xbias} \right)} \\{f_{y}\left( {t_{y} + t_{ybias}} \right)}\end{bmatrix}}}$and can be collected together for multiple observations of the samepallet location 2704 to provide a series of equations that can be solvedfor

$\quad\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w}\end{bmatrix}$in a manner similar to that of Equation 28A.

For simplicity, the different rotation matrices of Equations 21, 29D,and 30D can be conceptualized as Equation 30E where:

$\left\lbrack R_{cam} \right\rbrack = \begin{bmatrix}r_{00} & r_{01} & r_{02} \\r_{10} & r_{11} & r_{12} \\r_{20} & r_{21} & r_{22}\end{bmatrix}$

Substituting the notation of Equation 30E into Equation 20A and assumingthat

${\left\lbrack T_{cam} \right\rbrack = \begin{bmatrix}t_{xbias} \\\left( {t_{y_{n}} + t_{ybias}} \right) \\0\end{bmatrix}},$a pair of linear equations analogous to those of Equation 25A can bedetermined for any rotation matrix [R_(cam)]. This pair of linearequations can be solved for the pallet location (X_(w), Y_(w), Z_(w))and are given by Equation 30F:

${\begin{bmatrix}a_{n} & b_{n} & c_{n} \\d_{n} & e_{n} & f_{n}\end{bmatrix}\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w}\end{bmatrix}} = \begin{bmatrix}{f_{x}\left( t_{xbias} \right)} \\{f_{y}\left( {t_{y_{n}} + t_{ybias}} \right)}\end{bmatrix}$ where: a_(n) = x_(col_(n))r₂₀ − f_(x)r₀₀ − x₀r₂₀b_(n) = x_(col_(n))r₂₁ − f_(x)r₀₁ − x₀r₂₁c_(n) = x_(col_(n))r₂₂ − f_(x)r₀₂ − x₀r₂₂d_(n) = y_(row_(n))r₂₀ − f_(y)r₁₀ − y₀r₂₀e_(n) = y_(row_(n))r₂₁ − f_(y)r₁₁ − y₀r₂₁f_(n) = y_(row_(n))r₂₂ − f_(y)r₁₂ − y₀r₂₂

Similar to before, multiple observations of a pallet location can becollected and used to solve for the pallet location (X_(w), Y_(w),Z_(w)) in a least squares manner. Thus, using this notation, an equationanalogous to Equation 28A can be calculated according to:

$\begin{matrix}{\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w}\end{bmatrix} = {{{{\left( {\lbrack A\rbrack^{T}\lbrack A\rbrack} \right)^{- 1}\lbrack A\rbrack}^{T}\lbrack B\rbrack}\mspace{14mu}{\text{where:}\lbrack A\rbrack}} = {{\begin{bmatrix}a_{1} & b_{1} & c_{1} \\d_{1} & e_{1} & f_{1} \\a_{2} & b_{2} & c_{2} \\d_{2} & e_{2} & f_{2} \\a_{3} & b_{3} & c_{3} \\d_{3} & e_{3} & f_{3} \\\vdots & \vdots & \vdots\end{bmatrix}\mspace{14mu}{{and}\mspace{14mu}\lbrack B\rbrack}} = \begin{bmatrix}{f_{x}t_{xbias}} \\{f_{y}\left( {t_{y_{1}} + t_{ybias}} \right)} \\{f_{x}t_{xbias}} \\{f_{y}\left( {t_{y_{2}} + t_{ybias}} \right)} \\{f_{x}t_{xbias}} \\{f_{y}\left( {t_{y_{3}} + t_{ybias}} \right)} \\\vdots\end{bmatrix}}}} & {{Equation}\mspace{14mu} 31\; A}\end{matrix}$

The elevation angle, θ_(e), (or in the more general full rotationmatrix, the angle θ_(x)) which in the above equations, represents therotation of the imaging camera axis about the X-axis may include morethan one component that contributes to the overall angle. For example,the elevation angle θ_(e) may include a first component that correspondsto a fixed rotation angle θ_(ec) based on an angle at which the imagingcamera 130 is mounted on the fork carriage apparatus 40 and a secondcomponent that corresponds to an angle θ_(ef) at which the vehicle forks42A and 42B may be tilted relative to the carriage support structure 44Avia the tilt piston/cylinder unit 44D. Thus, in the above equations,θ_(e)=(θ_(ec)+θ_(ef)).

The contribution of the fork elevation angle θ_(ef) to the overallelevation angle θ_(e) can be measured by a conventional tilt anglesensor (not shown) associated with one or both of the carriage supportstructure 44A and the fork carriage frame 44B. This conventional tiltangle sensor may have a resolution of about 0.001 degrees and providethe tilt angle of the forks 42A, 42B to the vehicle computer 50, whichcan communicate this information to the image analysis computer 110. Thecontribution of the camera elevation angle θ_(ec) to the overallelevation angle θ_(e) can be determined by an extrinsic calibrationroutine, as explained in detail below.

If both components of angle θ_(e) (i.e., θ_(ec) and θ_(ef)) are constantvalues, then determining and using the elevation angle θ_(e) for any ofthe above equations is relatively straightforward. In certainembodiments, the imaging camera 130 is mounted on the fork carriageapparatus 40 so that its elevation angle θ_(ec) oes not change betweenimage frames. However, the elevation angle component due to tilting ofthe forks, θ_(ef), may change between image frames. Under thesecircumstances, the rotation matrices discussed above can be manipulatedto account for the potential variability of θ_(ef).

Referring back to Equation 21, one example rotation matrix is providedby:

$\left\lbrack R_{{cam}_{1}} \right\rbrack = \begin{bmatrix}1 & 0 & 0 \\0 & {\cos\;\theta_{e}} & {{- \sin}\;\theta_{e}} \\0 & {\sin\;\theta_{e}} & {\cos\;\theta_{e}}\end{bmatrix}$where, as mentioned, θ_(e)=(θ_(ec)+θ_(ef)). An intermediate rotationmatrix can be constructed that only accounts for the contribution of thecamera angle θ_(ec) according to:

$\left\lbrack R_{{cam}_{1}}^{\prime} \right\rbrack = \begin{bmatrix}1 & 0 & 0 \\0 & {\cos\;\theta_{ec}} & {{- \sin}\;\theta_{ec}} \\0 & {\sin\;\theta_{ec}} & {\cos\;\theta_{ec}}\end{bmatrix}$

Using this intermediate rotation matrix, an intermediate extrinsiccamera matrix can be written as:

$\left\lbrack R_{{cam}_{1}}^{\prime} \middle| T_{cam} \right\rbrack = \begin{bmatrix}1 & 0 & 0 & t_{xbias} \\0 & {\cos\;\theta_{ec}} & {{- \sin}\;\theta_{ec}} & \left( {t_{y} + t_{ybias}} \right) \\0 & {\sin\;\theta_{ec}} & {\cos\;\theta_{ec}} & 0\end{bmatrix}$One way to modify this intermediate extrinsic camera matrix is topost-multiply by a rotation matrix that accounts only for thecontribution of the fork tilt angle θ_(ef). To perform thismultiplication, the typical 3×3 rotation matrix can be expanded to a 4×4matrix according to:

$\quad\begin{bmatrix}1 & 0 & 0 & 0 \\0 & {\cos\;\theta_{ef}} & {{- \sin}\;\theta_{ef}} & 0 \\0 & {\sin\;\theta_{ef}} & {\cos\;\theta_{ef}} & 0 \\0 & 0 & 0 & 1\end{bmatrix}$and then performing the post-multiplication described above:

$\left\lbrack R_{{cam}_{1}}^{\prime} \middle| T_{cam} \right\rbrack \times \begin{bmatrix}1 & 0 & 0 & 0 \\0 & {\cos\;\theta_{ef}} & {{- \sin}\;\theta_{ef}} & 0 \\0 & {\sin\;\theta_{ef}} & {\cos\;\theta_{ef}} & 0 \\0 & 0 & 0 & 1\end{bmatrix}$results in Equation 31B:

$\quad\begin{bmatrix}1 & 0 & 0 & t_{xbias} \\0 & {{\cos\;\theta_{ec}\cos\;\theta_{ef}} - {\sin\;\theta_{ec}\sin\;\theta_{ef}}} & {{{- \cos}\;\theta_{ec}\sin\;\theta_{ef}} - {\sin\;\theta_{ec}\cos\;\theta_{ef}}} & \left( {t_{y} + t_{ybias}} \right) \\0 & {{\sin\;\theta_{ec}\cos\;\theta_{ef}} + {\cos\;\theta_{ec}\sin\;\theta_{ef}}} & {{{- \sin}\;\theta_{ec}\sin\;\theta_{ef}} + {\cos\;\theta_{ec}\cos\;\theta_{ef}}} & 0\end{bmatrix}$Using the two geometrical identities:cos(a+b)=cos(a)cos(b)−sin(a)sin(b)sin(a+b)=sin(a)cos(b)+cos(a)sin(b)Equation 31B can be simplified to:

$\quad\begin{bmatrix}1 & 0 & 0 & t_{xbias} \\0 & {\cos\left( {\theta_{ec} + \theta_{ef}} \right)} & {- {\sin\left( {\theta_{ec} + \theta_{ef}} \right)}} & \left( {t_{y} + t_{ybias}} \right) \\0 & {\sin\left( {\theta_{ec} + \theta_{ef}} \right)} & {\cos\left( {\theta_{ec} + \theta_{ef}} \right)} & 0\end{bmatrix}$Which can be further simplified to be:

$\quad\begin{bmatrix}1 & 0 & 0 & t_{xbias} \\0 & {\cos\;\theta_{e}} & {{- \sin}\;\theta_{e}} & \left( {t_{y} + t_{ybias}} \right) \\0 & {\sin\;\theta_{e}} & {\cos\;\theta_{e}} & 0\end{bmatrix}$which is simply [R_(cam) ₁ |T_(cam)].This technique may be used, in general, with any of the extrinsic cameramatrices discussed above so that any extrinsic camera matrix which isformed using a rotation matrix based on θ_(e) (e.g., Equations 21, 29D,30D or 30E) can be constructed from a respective intermediate extrinsiccamera matrix using an intermediate rotation matrix based on θ_(e) thatis then post-multiplied by a rotation matrix that is based on θ_(ef).

As mentioned earlier, the amount that the forks 42A and 42B tilt (i.e.,θ₀) may vary between each image frame that is acquired. Thus, theelevation angle θ_(e) (or more generally θ_(x)) can vary between imageframes as well. Accordingly, Equations 25A and 26A can be modified toaccount for the variability of the elevation angle θ_(e), as shownbelow:

$\begin{matrix}{\begin{bmatrix}{- f_{x}} & {\left( {x_{{col}_{n}} - x_{0}} \right)\sin\;\theta_{e_{n}}} & {\left( {x_{{col}_{n}} - x_{0}} \right)\cos\;\theta_{e_{n}}} \\0 & {{\left( {y_{{row}_{n}} - y_{0}} \right)\sin\;\theta_{e_{n}}} - {f_{y}\cos\;\theta_{e_{n}}}} & {{\left( {y_{{row}_{n}} - y_{0}} \right)\cos\;\theta_{e_{n}}} + {f_{y}\sin\;\theta_{e_{n}}}}\end{bmatrix}{\quad{\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w}\end{bmatrix} = {\begin{bmatrix}{f_{x}t_{xbias}} \\{f_{y}\left( {t_{y_{n}} + t_{ybias}} \right)}\end{bmatrix}{and}}}}} & {{Equation}\mspace{14mu} 25\; B} \\{\begin{bmatrix}{- f_{x}} & {\left( {x_{{col}_{1}} - x_{0}} \right)\sin\;\theta_{e_{1}}} & {\left( {x_{{col}_{1}} - x_{0}} \right)\cos\;\theta_{e_{1}}} \\0 & {{\left( {y_{{row}_{1}} - y_{0}} \right)\sin\;\theta_{e_{1}}} - {f_{y}\cos\;\theta_{e_{1}}}} & {{\left( {y_{{row}_{1}} - y_{0}} \right)\cos\;\theta_{e_{1}}} + {f_{y}\sin\;\theta_{e_{1}}}} \\{- f_{x}} & {\left( {x_{{col}_{2}} - x_{0}} \right)\sin\;\theta_{e_{2}}} & {\left( {x_{{col}_{2}} - x_{0}} \right)\cos\;\theta_{e_{2}}} \\0 & {{\left( {y_{{row}_{2}} - y_{0}} \right)\sin\;\theta_{e_{2}}} - {f_{y}\cos\;\theta_{e_{2}}}} & {{\left( {y_{{row}_{2}} - y_{0}} \right)\cos\;\theta_{e_{2}}} + {f_{y}\sin\;\theta_{e_{2}}}} \\{- f_{x}} & {\left( {x_{{col}_{3}} - x_{0}} \right)\sin\;\theta_{e_{3}}} & {\left( {x_{{col}_{3}} - x_{0}} \right)\cos\;\theta_{e_{3}}} \\0 & {{\left( {y_{{row}_{3}} - y_{0}} \right)\sin\;\theta_{e_{3}}} - {f_{y}\cos\;\theta_{e_{3}}}} & {{\left( {y_{{row}_{3}} - y_{0}} \right)\cos\;\theta_{e_{3}}} + {f_{y}\sin\;\theta_{e_{3}}}} \\\vdots & \vdots & \vdots\end{bmatrix}{\quad{\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w}\end{bmatrix} = \begin{bmatrix}{f_{x}t_{xbias}} \\{f_{y}\left( {t_{y_{1}} + t_{ybias}} \right)} \\{f_{x}t_{xbias}} \\{f_{y}\left( {t_{y_{2}} + t_{ybias}} \right)} \\{f_{x}t_{xbias}} \\{f_{y}\left( {t_{y_{3}} + t_{ybias}} \right)} \\\vdots\end{bmatrix}}}} & {{Equation}\mspace{14mu} 26\; B}\end{matrix}$

However, even with the variability of θ_(e) that can occur between imageframes, the technique for solving for pallet location (X_(w), Y_(w),Z_(w)) remains the same. Although, for each image frame, the imageanalysis computer 110 may also now store θ_(e) _(n) in addition to(x_(col) _(n) , y_(row) _(n) ) and t_(y) _(n) . Accordingly, equations25B and 26B can be rearranged to provide Equation 28B:

$\mspace{79mu}{\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w}\end{bmatrix} = {{\left( {\left\lbrack A^{\prime} \right\rbrack^{T}\left\lbrack A^{\prime} \right\rbrack} \right)^{- 1}\left\lbrack A^{\prime} \right\rbrack}^{T}\lbrack B\rbrack}}$$\mspace{79mu}{{{where}{\text{:}\left\lbrack A^{\prime} \right\rbrack}} = {\quad{{\begin{bmatrix}{- f_{x}} & {\left( {x_{{col}_{1}} - x_{0}} \right)\sin\;\theta_{e_{1}}} & {\left( {x_{{col}_{1}} - x_{0}} \right)\cos\;\theta_{e_{1}}} \\0 & {{\left( {y_{{row}_{1}} - y_{0}} \right)\sin\;\theta_{e_{1}}} - {f_{y}\cos\;\theta_{e_{1}}}} & {{\left( {y_{{row}_{1}} - y_{0}} \right)\cos\;\theta_{e_{1}}} + {f_{y}\sin\;\theta_{e_{1}}}} \\{- f_{x}} & {\left( {x_{{col}_{2}} - x_{0}} \right)\sin\;\theta_{e_{2}}} & {\left( {x_{{col}_{2}} - x_{0}} \right)\cos\;\theta_{e_{2}}} \\0 & {{\left( {y_{{row}_{2}} - y_{0}} \right)\sin\;\theta_{e_{2}}} - {f_{y}\cos\;\theta_{e_{2}}}} & {{\left( {y_{{row}_{2}} - y_{0}} \right)\cos\;\theta_{e}} + {f_{y}\sin\;\theta_{e_{2}}}} \\{- f_{x}} & {\left( {x_{{col}_{3}} - x_{0}} \right)\sin\;\theta_{e_{3}}} & {\left( {x_{{col}_{3}} - x_{0}} \right)\cos\;\theta_{e_{3}}} \\0 & {{\left( {y_{{row}_{3}} - y_{0}} \right)\sin\;\theta_{e_{3}}} - {f_{y}\cos\;\theta_{e_{3}}}} & {{\left( {y_{{row}_{3}} - y_{0}} \right)\cos\;\theta_{e}} + {f_{y}\sin\;\theta_{e_{3}}}} \\\vdots & \vdots & \vdots\end{bmatrix}\mspace{79mu}{{and}\mspace{79mu}\lbrack B\rbrack}} = \begin{bmatrix}{f_{x}t_{xbias}} \\{f_{y}\left( {t_{y_{1}} + t_{ybias}} \right)} \\{f_{x}t_{xbias}} \\{f_{y}\left( {t_{y_{2}} + t_{ybias}} \right)} \\{f_{x}t_{xbias}} \\{f_{y}\left( {t_{y_{3}} + t_{ybias}} \right)} \\\vdots\end{bmatrix}}}}$

In the above examples, one assumption was that the movement of the forkst_(y) occurs substantially in the direction of the Y-axis 2711 of theworld coordinate system. However, the mast assembly 30 may notnecessarily align with the direction of the world coordinate systemY-axis 2711. As shown in FIG. 27F, a vector 2775 may represent adirection of the mast assembly 30 with respect to the world coordinatesystem.

Thus, in FIG. 27F, there is a vector {right arrow over (V)}_(mast) 2776that represents a position 2770 of the forks 42A and 42B, along thevector direction 2775, in world system coordinates relative to the worldcoordinate origin 2706. The position 2770 of the forks 42A and 42B canbe considered as an origin of a fork coordinate system that along withthree orthogonal axes 2707A, 2711A, and 2705A define a fork coordinatesystem. Within the fork coordinate system a vector {right arrow over(V)}_(forks) 2774 represents a location 2704 of a pallet in fork systemcoordinates relative to the fork location 2770. As discussed before,there is a camera location 2708 that defines an origin of a cameracoordinate system that along with three orthogonal axes 2707B, 2711B,and 2705B define a camera coordinate system. Accordingly, there is avector {right arrow over (V)}_(cam) 2778 that represents the location2704 of the pallet in camera system coordinates relative to the cameralocation 2708.

The axes 2707B, 2711B, 2705B of the camera coordinate system may berotated relative to the axes 2707A, 2711A, 2705A of the fork coordinatesystem and may also be translated relative to the fork coordinate systemas well. Thus, the pallet location 2704 in the fork coordinate systemcan be transformed into a location in the camera coordinate systemaccording to Equation 15B:

$\begin{bmatrix}X_{c} \\Y_{c} \\Z_{c}\end{bmatrix} = {{\left\lbrack R_{{cam} - {forks}} \right\rbrack\begin{bmatrix}X_{f} \\Y_{f} \\Z_{f}\end{bmatrix}} + \begin{bmatrix}{tcf}_{xbias} \\{tcf}_{ybias} \\{tcf}_{zbias}\end{bmatrix}}$

where:

${\overset{->}{V}}_{cam} = \begin{bmatrix}X_{c} \\Y_{c} \\Z_{c}\end{bmatrix}$the coordinates of the pallet location 2704 in camera systemcoordinates;

$\left\lbrack R_{{cam} - {forks}} \right\rbrack = \begin{bmatrix}{rcf}_{00} & {rcf}_{01} & {rcf}_{02} \\{rcf}_{10} & {rcf}_{11} & {rcf}_{12} \\{rcf}_{20} & {rcf}_{21} & {rcf}_{22}\end{bmatrix}$a rotation matrix that represents the rotation of the camera coordinatesystem relative to the fork coordinate system;

${\overset{->}{V}}_{forks} = \begin{bmatrix}X_{f} \\Y_{f} \\Z_{f}\end{bmatrix}$the coordinates of the pallet location 2704 in the fork coordinatesystem; and

$\quad\begin{bmatrix}{tcf}_{xbias} \\{tcf}_{ybias} \\{tcf}_{zbias}\end{bmatrix}$is a translation vector representing a displacement of the cameracoordinate system origin 2708 relative to the fork coordinate systemorigin 2770 along the direction of the axes 2707A, 2711A, and 2705A ofthe fork coordinate system.

The location of the fork coordinate system origin 2770 is providedaccording to Equation 15C:

${\overset{->}{V}}_{mast} = {\left\lbrack R_{{mast} - {world}} \right\rbrack\begin{bmatrix}0 \\t_{y} \\0\end{bmatrix}}$

where:

-   -   {right arrow over (V)}_(mast) is the location, in world system        coordinates, of the fork location 2770;

$\left\lbrack R_{{mast} - {world}} \right\rbrack = \begin{bmatrix}{rmw}_{00} & {rmw}_{01} & {rmw}_{02} \\{rmw}_{10} & {rmw}_{11} & {rmw}_{12} \\{rmw}_{20} & {rmw}_{21} & {rmw}_{22}\end{bmatrix}$a rotation matrix that represents the rotation of the mast assembly 30relative to the world coordinate system axes 2707, 2711, and 2705; and

$\quad\begin{bmatrix}0 \\t_{y} \\0\end{bmatrix}$is a translation vector representing the displacement of the forks 42Aand 42B from the world coordinate origin 2706 in a direction along thevector 2775.

If the axes 2707A, 2711A, and 2705A of the fork coordinate system arerotationally aligned with the axes 2707, 2711, and 2705 of the worldcoordinate system, then the vector {right arrow over (V)}_(world) 2772is, by vector addition, given according to Equation 15D:{right arrow over (V)} _(world) ={right arrow over (V)} _(mast) +{rightarrow over (V)} _(forks)

Multiplying both sides of Equation 15D by [R_(cam−forks)] provides:[R _(cam−forks) ]{right arrow over (V)} _(world) =[R _(cam−forks)]{right arrow over (V)} _(mast) +[R _(cam−forks) ]{right arrow over (V)}_(forks)

which, by substituting in Equation 15C, provides:

${\left\lbrack R_{{cam} - {forks}} \right\rbrack{\overset{->}{V}}_{world}} = {{{\left\lbrack R_{{cam} - {forks}} \right\rbrack\left\lbrack R_{{mast} - {world}} \right\rbrack}\begin{bmatrix}0 \\t_{y} \\0\end{bmatrix}} + {\left\lbrack R_{{cam} - {forks}} \right\rbrack{\overset{->}{V}}_{forks}}}$

The terms of Equation 15B can be rearranged to provide that:

${\begin{bmatrix}X_{c} \\Y_{c} \\Z_{c}\end{bmatrix} - \begin{bmatrix}{tcf}_{xbias} \\{tcf}_{ybias} \\{tcf}_{zbias}\end{bmatrix}} = {\left\lbrack R_{{cam} - {forks}} \right\rbrack\begin{bmatrix}X_{f} \\Y_{f} \\Z_{f}\end{bmatrix}}$which can be substituted in the above equation to provide Equation 15E:

${\left\lbrack R_{{cam} - {forks}} \right\rbrack{\overset{->}{V}}_{world}} = {{{\left\lbrack R_{{cam} - {forks}} \right\rbrack\left\lbrack R_{{mast} - {world}} \right\rbrack}\begin{bmatrix}0 \\t_{y} \\0\end{bmatrix}} + \begin{bmatrix}X_{c} \\Y_{c} \\Z_{c}\end{bmatrix} - \begin{bmatrix}{tcf}_{xbias} \\{tcf}_{ybias} \\{tcf}_{zbias}\end{bmatrix}}$

The terms in Equation 15E can be rearranged to produce Equation 15F asfollows:

${{\left\lbrack R_{{cam} - {forks}} \right\rbrack{\overset{->}{V}}_{world}} - {{\left\lbrack R_{{cam} - {forks}} \right\rbrack\left\lbrack R_{{mast} - {world}} \right\rbrack}\begin{bmatrix}0 \\t_{y} \\0\end{bmatrix}} + \begin{bmatrix}{tcf}_{xbias} \\{tcf}_{ybias} \\{tcf}_{zbias}\end{bmatrix}} = \begin{bmatrix}X_{c} \\Y_{c} \\Z_{c}\end{bmatrix}$

The two 3×3 rotation matrices [R_(cam-forks)] and [R_(mast-world)]multiply together to form a composite 3×3 rotation matrix with elementswhose values depend on both

$\begin{bmatrix}{rcf}_{00} & {rcf}_{01} & {rcf}_{02} \\{rcf}_{10} & {rcf}_{11} & {rcf}_{12} \\{rcf}_{20} & {rcf}_{21} & {rcf}_{22}\end{bmatrix}$ ${{and}\begin{bmatrix}{rmw}_{00} & {rmw}_{01} & {rmw}_{02} \\{rmw}_{10} & {rmw}_{11} & {rmw}_{12} \\{rmw}_{20} & {rmw}_{21} & {rmw}_{22}\end{bmatrix}}.$However, when multiplied out with the translation vector

$\begin{bmatrix}0 \\t_{y} \\0\end{bmatrix},$the result is a 3×1 translation vector that can be conceptualized as

$\quad\begin{bmatrix}{\alpha_{r}t_{y}} \\{\beta_{r}t_{y}} \\{\gamma_{r}t_{y}}\end{bmatrix}$where the values for α_(r), β_(r), and γ_(r) depend on the elements ofthe composite rotation matrix. Thus, Equation 15F can be rearranged toprovide Equation 15G:

$\begin{bmatrix}X_{c} \\Y_{c} \\Z_{c}\end{bmatrix} = {{\left\lbrack R_{{cam} - {forks}} \right\rbrack\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w}\end{bmatrix}} - \begin{bmatrix}{\alpha_{r}t_{y}} \\{\beta_{r}t_{y}} \\{\gamma_{r}t_{y}}\end{bmatrix} + \begin{bmatrix}{tcf}_{xbias} \\{tcf}_{ybias} \\{tcf}_{zbias}\end{bmatrix}}$

where:

${\overset{\rightarrow}{V}}_{world} = \begin{bmatrix}X_{w} \\Y_{w} \\Z_{w}\end{bmatrix}$is the location in world system coordinates of the pallet location 2704.

Incorporating the negative sign into the values α_(r), β_(r), and γ_(r)allows Equation 15G to be written more simply as Equation 15H:

$\begin{bmatrix}X_{c} \\Y_{c} \\Z_{c}\end{bmatrix} = {{\left\lbrack R_{{cam} - {forks}} \right\rbrack\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w}\end{bmatrix}} + \begin{bmatrix}{{\alpha_{r}t_{y}} + {tcf}_{xbias}} \\{{\beta_{r}t_{y}} + {tcf}_{ybias}} \\{{\gamma_{r}t_{y}} + {tcf}_{zbias}}\end{bmatrix}}$

which is similar in form to the original Equations 15A and 17A discussedabove. Thus, similar to Equation 17A, Equation 15H can be rewritten inhomogenous coordinates as Equation 17B:

$\begin{bmatrix}X_{c} \\Y_{c} \\Z_{c}\end{bmatrix} = {\begin{bmatrix}\; & {{\alpha_{r}t_{y}} + {tcf}_{xbias}} \\\left\lbrack R_{{cam} - {forks}} \right\rbrack & {{\beta_{r}t_{y}} + {tcf}_{ybias}} \\\; & {{\gamma_{r}t_{y}} + {tcf}_{zbias}}\end{bmatrix}\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w} \\1\end{bmatrix}}$

Thus, Equation 17B represents an alternative way to calculate cameracoordinates

$\quad\begin{bmatrix}X_{c} \\Y_{c} \\Z_{c}\end{bmatrix}$of the pallet location 2704. This alternative method takes into accountthat the displacement t_(y) along the direction of the vector 2775(e.g., the mast assembly 30) does not necessarily occur in the samedirection as the Y-axis 2711 of the world coordinate system. Similar toEquation 19A discussed above, camera coordinates in accordance with thisalternative method are related to image pixel locations according toEquation 19B:

${\lambda\begin{bmatrix}x_{col} \\y_{row} \\1\end{bmatrix}} = {{\lbrack K\rbrack\begin{bmatrix}\; & {{\alpha_{r}t_{y}} + {tcf}_{xbias}} \\\left\lbrack R_{{cam} - {forks}} \right\rbrack & {{\beta_{r}t_{y}} + {tcf}_{ybias}} \\\; & {{\gamma_{r}t_{y}} + {tcf}_{zbias}}\end{bmatrix}}\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w} \\1\end{bmatrix}}$${{where}\mspace{14mu}\lbrack K\rbrack} = \begin{bmatrix}f_{x} & 0 & x_{0} \\0 & f_{y} & y_{0} \\0 & 0 & 1\end{bmatrix}$

Parameter estimation practicalities suggest some changes may be made toEquation 19B. A non-zero value for tcf_(zbias) may imply a multiplicityof solutions. Second, it has been observed that attempts to estimateβ_(r) tend to produce a trivial solution of all zeros. Hence, twoassumptions may be made that help simplify Equation 19B: first, thevalue of tcf_(zbias)=0 and, secondly, β_(r)=1. Using these assumptions,Equation 19B can be written as Equation 19C:

${\lambda\begin{bmatrix}x_{col} \\y_{row} \\1\end{bmatrix}} = {{\lbrack K\rbrack\begin{bmatrix}\; & {{\alpha_{r}t_{y}} + {tcf}_{xbias}} \\\left\lbrack R_{{cam} - {forks}} \right\rbrack & {t_{y} + {tcf}_{ybias}} \\\; & {\gamma_{r}t_{y}}\end{bmatrix}}\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w} \\1\end{bmatrix}}$

When solved, the particular form of Equation 19C shown above results inheight estimates, i.e., values for Y_(W), that have an appropriatemagnitude but are negative in sign. Thus, other algorithms and methodsthat perform calculations using a Y_(W) value can be appropriatelydesigned to account for the negatively signed Y_(W) values. Equation 19Crelates the pallet location 2704, up to a scale factor λ, with a grayscale image pixel location (x_(col), y_(row)). This implies that theleft and right sides of Equation 19C are collinear vectors which resultsin a zero cross-product. The cross product equation, similar to Equation20A is:

$\begin{matrix}{{\begin{bmatrix}x_{col} \\y_{row} \\1\end{bmatrix} \times {{\lbrack K\rbrack\begin{bmatrix}\; & {{\alpha_{r}t_{y}} + {tcf}_{xbias}} \\\left\lbrack R_{{cam} - {forks}} \right\rbrack & {t_{y} + {tcf}_{ybias}} \\\; & {\gamma_{r}t_{y}}\end{bmatrix}}\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w} \\1\end{bmatrix}}} = 0} & {{Equation}\mspace{14mu} 20B}\end{matrix}$

Similar to the steps discussed above, the cross-product equation ofEquation 20B provides a pair of linear equations that can be collectedfor a number of observations and solved for the pallet location (X_(w),Y_(w), Z_(w)). For a single observation, n, of a pallet when thedisplacement of the forks 42A and 42B is t_(y) _(n) , the two linearequations are given by Equation 30G:

${\begin{bmatrix}a_{n} & b_{n} & c_{n} \\d_{n} & e_{n} & f_{n}\end{bmatrix}\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w}\end{bmatrix}} = \begin{bmatrix}{{f_{x}\left( {{tcf}_{xbias} + {\alpha_{r}t_{y_{n}}}} \right)} - {\gamma_{r}{t_{y_{n}}\left( {x_{{col}_{n}} - x_{0}} \right)}}} \\{{f_{y}\left( {t_{y_{n}} + {tcf}_{ybias}} \right)} - {\gamma_{r}{t_{y_{n}}\left( {y_{{row}_{n}} - y_{0}} \right)}}}\end{bmatrix}$ where : a_(n) = x_(col_(n))rcf₂₀ − f_(x)rcf₀₀ − x₀rcf₂₀b_(n) = x_(col_(n))rcf₂₁ − f_(x)rcf₀₁ − x₀rcf₂₁c_(n) = x_(col_(n))rcf₂₂ − f_(x)rcf₀₂ − x₀rcf₂₂d_(n) = y_(row_(n))rcf₂₀ − f_(y)rcf₁₀ − y₀rcf₂₀e_(n) = y_(row_(n))rcf₂₁ − f_(y)rcf₁₁ − y₀rcf₂₁f_(n) = y_(row_(n))rcf₂₂ − f_(y)rcf₁₂ − y₀rcf₂₂

Similar to before, multiple observations of a pallet location can becollected and used to solve for the pallet location (X_(w), Y_(w),Z_(w)) in a least squares manner.

The above methods for solving for the pallet location (X_(w), Y_(w),Z_(w)) may be susceptible to error in the sense that one bad observationmay undesirably skew the least-squares estimate. Thus, in a set ofobservations (x_(col) _(n) , y_(row) _(n) ) for a pallet location 2704there may be one or more observations that should not be considered whensolving Equation 28A (or Equation 28B or Equation 31A) for the palletlocation. RANSAC is an abbreviation for “RANdomSAmple Consensus” whichis an iterative method to estimate parameters of a mathematical modelfrom a set of observed data which contains outliers. A basic assumptionis that the data consists of “inliers” which are data whose distributioncan be explained by some set of model parameters, and “outliers” whichare data that do not fit the model. In addition to this, the data can besubject to noise. The outliers can come, for example, from extremevalues of the noise or from erroneous measurements or incorrecthypotheses about the interpretation of data.

FIG. 27D depicts a flowchart of an exemplary method of applying RANSACto the multiple observation image pixel locations used to determine apallet location (X_(w), Y_(w), Z_(w)). Each iteration of the RANSACmethod selects a minimum sample set which is then used to identify arespective consensus set of data. By repeating such iterations fordifferent minimum sample sets, more than one consensus set can beidentified. Ultimately, the largest consensus set from among thedifferent consensus sets can then be used when performing subsequentcalculations.

For a scored candidate object in the ExistingObjectlist, there will be anumber of different observation image pixel locations (x_(col) _(n) ,y_(row) _(n) ) collected from n image frames associated with thatparticular scored candidate object in the ExistingObjectlist, whereinthe actual pixel location corrected for image distortion, as discussedbelow, is stored in the measurement list of the ExistingObjectlist. Instep 2730, the image analysis computer 110 determines if there are moreimage pixel locations to test or if a predetermined maximum number ofiterations have been performed already. Initially, all the differentobservation image pixel locations, i.e., which comprise actual palletpixel coordinates corrected for image distortion, are available forselection, so the process continues with step 2732. In step 2732, theimage analysis computer 110 randomly selects two of these observationimage pixel locations, which can be referred to as a minimum sample set(MSS). The two observation image pixel locations are flagged so thatthey are not used again in a subsequent iteration. Using the MSS, theimage analysis computer 110 solves Equation 28A (or equation 28B orequation 31A), in a least squares manner, for (X_(w), Y_(w), Z_(w)), instep 2734. For example, if respective image pixel locations from grayscale images n=3 and n=6 are randomly selected, then Equation 26B isrealized as:

$\begin{bmatrix}{- f_{x}} & {\left( {x_{{col}_{3}} - x_{0}} \right)\sin\;\theta_{e_{3}}} & {\left( {x_{{col}_{3}} - x_{0}} \right)\cos\;\theta_{e_{3}}} \\0 & {{\left( {y_{{row}_{3}} - y_{0}} \right)\sin\;\theta_{e_{3}}} - {f_{y}\cos\;\theta_{e_{3}}}} & {{\left( {y_{{row}_{3}} - y_{0}} \right)\cos\;\theta_{e_{3}}} + {f_{y}\sin\;\theta_{e_{3}}}} \\{- f_{x}} & {\left( {x_{{col}_{6}} - x_{0}} \right)\sin\;\theta_{e_{6}}} & {\left( {x_{{col}_{6}} - x_{0}} \right)\cos\;\theta_{e_{6}}} \\0 & {{\left( {y_{{row}_{6}} - y_{0}} \right)\sin\;\theta_{e_{6}}} - {f_{y}\cos\;\theta_{e_{6}}}} & {{\left( {y_{{row}_{6}} - y_{0}} \right)\cos\;\theta_{e_{6}}} + {f_{y}\sin\;\theta_{e_{6}}}}\end{bmatrix}{\quad{\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w}\end{bmatrix} = \begin{bmatrix}{f_{x}t_{xbias}} \\{f_{y}\left( {t_{y_{3}} + t_{ybias}} \right)} \\{f_{x}t_{xbias}} \\{f_{y}\left( {t_{y_{6}} + t_{ybias}} \right)}\end{bmatrix}}}$and can be used, as shown in Equation 28B, for example, to estimate aleast-squares solution for (X_(w), Y_(w), Z_(w)). This solution can bereferred to as the “MSS model” as it is the estimate for (X_(w), Y_(w),Z_(w)) using only the MSS. In step 2736, an error value is calculatedthat is based on the error between each image pixel location in the MSSand the MSS model. In particular, the squared error of each observationimage pixel location with the MSS model is calculated. For example, thesquared error, ε_(n), where rotation of the camera coordinate systemabout only the X-axis is contemplated, may be calculated according to:ε_(n)=(Z _(w)(x _(col) _(n) −x ₀)cos θ_(e) _(n) +Y _(w)(x _(col) _(n) −x₀)sin θ_(e) _(n) −X _(w) f _(x) −f _(x) t _(xbias))²+(Z _(w)((y _(row)_(n) −y ₀)cos θ_(e) _(n) +f _(y) sin θ_(e) _(n) )+Y _(w)((y _(row) _(n)−y ₀)sin θ_(e) _(n) −f _(y) cos θ_(e) _(n) )−f _(y)(t _(y) _(n) +t_(ybias)))²  Equation 32A:

In the more general case, where the conceptual rotation matrix [R_(cam)]of Equation 30E is used, then the squared error, ε_(n), may becalculated according to Equation 32B:ε_(n)=(a _(n) X _(w) +b _(n) Y _(w) +c _(n) Z _(w) −f _(x) t_(xbias))²+(d _(n) X _(w) +e _(n) Y _(w) +f _(n) Z _(w) −f _(y)(t _(y)_(n) +t _(ybias)))²

In the specific MSS of this example, two squared error values (i.e., ε₃and ε₆) are calculated in step 2736. Using these two squared errorvalues, the standard deviation, σ_(ε), for the two error values iscalculated and, in step 2738, the image analysis computer 110 calculatesa threshold value, δ, according to:δ=1.0σ_(ε)  Equation 33:

In step 2740, the image analysis computer 110 determines, for thisscored candidate object, a respective error for each of the remainingobservation image pixels locations not in the MSS by employing Equation32A (or Equation 32B) and the X_(w), Y_(w), Z_(w) values, i.e., the MSSmodel, determined by way of Equation 28A (or Equation 28B or Equation31A) using the MSS. The image analysis computer 110 then determines allof the observation image pixel locations, i.e., those not in the MSS,that have an error (according to Equation 32A or 32B) that is less thanthe threshold 6. The observation image pixel locations which areidentified in step 2740 are the consensus set for this particulariteration of the RANSAC method using this particular MSS. If thisconsensus set is larger than any previously determined consensus set,then it is stored in step 2742 as the largest consensus set. If thisconsensus set is not larger than any previously determined consensusset, then the largest consensus set that is stored remains stored.

The RANSAC method returns to step 2730 where the image analysis computer110 determines if there are any more observation image pixel locationsto try as the MSS or if the predetermined maximum number of iterationshas been performed. For example, the iterations may be limited to about20, if desired. Eventually, this threshold will be reached or all of theobservation image pixel locations will have been selected for inclusionin at least one MSS. Once that happens, the image analysis computer 110can use the observation image pixel locations in the largest consensusset to solve Equation 28A (or Equation 28B or Equation 31A) for (X_(w),Y_(w), Z_(w)) in step 2744. Thus, the solution for the pallet location2704 may not necessarily use “outlier” observation image pixel locationsthat would introduce unwanted error into the solution. However, theoutlier observation image pixel locations can remain in the record forthis scored candidate object in the ExistingObjectlist.

Once the pallet location 2704 is determined, the vehicle computer 50 cancontrol the placement of the forks 42A and 42B so that the forks canengage the pallet 2702, as discussed further below.

The above discussion about the projection of a pallet location 2704(X_(w), Y_(w), Z_(w)) to an observation image pixel location (x_(col),y_(row)) uses a simplified model that does not take into considerationdistortion that may be introduced by the lens of the imaging camera 130.Thus, a measured image pixel location identified using an actual imagingcamera 130 having lens distortion may be at a location 2723 (See FIG.27C), which is different than if the pallet location 2704 had beenprojected through an ideal lens undistorted onto the image plane 2712 atpoint 2718 as was used to develop the equations described above. Toaccount for the potential of lens distortion, the simplified projectionmodel can be expanded. In the simplified projection model, Equation 16can be rearranged to provide:x _(col) =f _(x) x″+x ₀ and y _(row) =f _(y) y″+y ₀  Equation 34:

-   -   where the undistorted camera coordinates are considered to be:

$x^{''} = {{\frac{X_{c}}{Z_{c}}\mspace{14mu}{and}\mspace{14mu} y^{''}} = \frac{Y_{c}}{Z_{c}}}$

The distortion of a lens may include tangential distortion coefficients(p₁, p₂) and radial distortion coefficients (k₁, k₂, k₃), whichcoefficients are defined for a particular camera lens as is well knownin the art. One common model to account for lens distortion is to extendthe original transformation model, Equation 34, according to:x _(col) =f _(x) x′+x ₀ and y _(row) =f _(y) y′+y ₀  Equation 35:

-   -   where: x′ and y′ are considered to be the distorted camera        coordinates and are related to the undistorted camera        coordinates x″, y″ according to:        x″=x′(1+k ₁ r ² +k ₂ r ⁴ +k ₃ r ⁶)+2p ₁ x′y′+p ₂(r ²+2x′ ²);        y″=y′(1+k ₁ r ² +k ₂ r ⁴ +k ₃ r ⁶)+2p ₂ x′y′+p ₁(r ²+2y′ ²); and        r ² =x′ ² +y′ ²

When identifying measured image pixel locations and when predictingwhere an image pixel location may be located in a subsequent imageframe, to be discussed below, some of the calculations may occur withinthe projection model that has been expanded to consider lens distortionand other calculations may occur within the simplified projection modelthat does not consider lens distortion. With the addition of theexpanded projection model there can now be two different types of imagepixel locations—undistorted image pixel coordinates and distorted imagepixel coordinates. Techniques for translating between distorted andundistorted values may simplify computations and calculations atdifferent steps in the image analysis processes as more fully describedbelow.

An actual gray scale image that is acquired by the imaging camera 130and analyzed by the image analysis computer 110 involves an actual lens,not an ideal lens. Thus, when the image analysis computer 110 measuresand analyzes image pixel locations for a gray scale image or othercalculated images derived from the gray scale image, the image pixelcoordinates may be in distorted image space. As mentioned above, it maybe beneficial to translate those pixel coordinates into the projectionmodel of Equation 34, which can be referred to as undistorted imagespace. To help distinguish between the pixel coordinates in the twodifferent types of image space, an additional modifier can be added tothe subscript for a coordinate location (x_(col), y_(row)). Coordinatesfor pixel locations in distorted image space (according to Equation 35)can be referred to as (x_(D-col),y_(D-row)) and the coordinates forpixel locations in undistorted image space (according to Equation 34)can be referred to as (x_(UD-col), y_(UD-row)).

To convert undistorted pixel coordinates (x_(UD-col)−y_(UD-row)) todistorted image space, Equation 34 is used to determine x″ and y″according to:x″=(x _(UD-col) −x ₀)/f _(x) and y″=(y _(UD-row) −y ₀)/f _(y)

using these values for x″ and y″, the terms x′ and y′ of Equation 35 arecalculated according to:x″=x′(1+k ₁ r ² +k ₂ r ⁴ +k ₃ r ⁶)+2p ₁ x′y′+p ₂(r ²+2x′ ²)y″=y′(1+k ₁ r ² +k ₂ r ⁴ +k ₃ r ⁶)+2p ₂ x′y′+p ₁(r ²+2y′ ²)The above two equations can be rearranged to produce:

$x^{\prime} = {\frac{1}{\left( {1 + {k_{1}r^{2}} + {k_{2}r^{4}} + {k_{3}r^{6}}} \right)}\left( {x^{''} - {2p_{1}x^{\prime}y^{\prime}} - {p_{2}\left( {r^{2} + {2x^{\prime 2}}} \right)}} \right)}$$y^{\prime} = {\frac{1}{\left( {1 + {k_{1}r^{2}} + {k_{2}r^{4}} + {k_{3}r^{6}}} \right)}\left( {y^{''} - {2p_{2}x^{\prime}y^{\prime}} - {p_{1}\left( {r^{2} + {2y^{\prime 2}}} \right)}} \right)}$Each of the above two equations cannot be solved directly but may besolved using an iterative approach that will converge to a respectivevalue for x′ and y′ wherein:

${x^{\prime}\left( {t + 1} \right)} = {\frac{1}{\left( {1 + {k_{1}r^{2}} + {k_{2}r^{4}} + {k_{3}r^{6}}} \right)}\left( {x^{''} - {2p_{1}{x^{\prime}(t)}{y^{\prime}(t)}} - {p_{2}\left( {r^{2} + {2{x^{\prime}(t)}^{2}}} \right)}} \right)}$${y^{\prime}\left( {t + 1} \right)} = {\frac{1}{\left( {1 + {k_{1}r^{2}} + {k_{2}r^{4}} + {k_{3}r^{6}}} \right)}\left( {y^{''} - {2p_{2}{x^{\prime}(t)}{y^{\prime}(t)}} - {p_{1}\left( {r^{2} + {2{y^{\prime}(t)}^{2}}} \right)}} \right)}$

where t is the index value denoting the iteration. Thus, the left handsides of the equations are calculated and used in the next iteration inthe right hand sides. The iterative process starts (i.e., t=0) with aninitial estimate for x′(0) and y′(0) and continues for a number ofiterations until the error between successive iterations is sufficientlysmall. As an example, the error can be the Euclidian distance betweentwo estimates for (x′, y′). Thus, the error can be calculated as:error=√{square root over (((x′(t+1)−x′(t))²+(y′(t+1)−y′(t))²))}{squareroot over (((x′(t+1)−x′(t))²+(y′(t+1)−y′(t))²))}{square root over(((x′(t+1)−x′(t))²+(y′(t+1)−y′(t))²))}{square root over(((x′(t+1)−x′(t))²+(y′(t+1)−y′(t))²))}The iterations continue until the error is smaller than a predeterminedthreshold value such as, for example, 0.0001.

In one example embodiment, the initial estimate values for the iterativeprocess can be: x′(0)=x″ and y′(0)=y″.

Once the values for x′ and y′ are calculated, they can be used inEquation 35 to determine the distorted location coordinates:x _(D-col) =f _(x) x′+x ₀ and y _(D-row) =f _(y) y′+y ₀.

Conversion of distorted image space pixel coordinates (x_(D-col),y_(D-row)) to undistorted image space pixel coordinates (x_(UD-col),y_(UD-row)) can be accomplished in a similar manner. First, Equation 35is used to determine x′ and y′ according to:x′=(x _(D-col) −x ₀)/f _(x) and y′=(y _(D-row) −y ₀)/f _(y)using these values for x′ and y′, the terms x″ and y″ of Equation 34 arecalculated according to:x″=x′(1+k ₁ r ² +k ₂ r ⁴ +k ₃ r ⁶)+2p ₁ x′y′+p ₂(r ²+2x′ ²)y″=y′(1+k ₁ r ² +k ₂ r ⁴ +k ₃ r ⁶)+2p ₂ x′y′+p ₁(r ²+2y′ ²)

Once the values for x″ and y″ are calculated, they can be used inEquation 34 to determine the undistorted location coordinates:x _(UD-col) =f _(x) x″+x ₀ and y _(UD-row) =f _(y) y″+y ₀

Example values for lens distortion coefficients, for a 480×752 pixelimage captured using a 5.5 mm focal length lens focused at infinity, canbe:k ₁=0.1260,k ₂=−1.0477,k ₃=2.3851,p ₁=−0.0056,p ₂=−0.0028

Example values for lens distortion coefficients, for a 240×376 pixelimage captured using a 5.5 mm focal length lens focused at infinity, canbe:k ₁=0.091604,k ₂=−0.235003,k ₃=0,p ₁=−0.003286,p ₂=−0.010984

One particular example of converting between distorted and undistortedimage space can involve the locating of pallet lower center points forscored candidate objects and then predicting their location andcalculating a corresponding prediction window in a subsequent imageframe.

Returning to FIG. 25, for each image frame in which the scored candidateobject is detected, a respective measured undistorted pixel locationvalue (x_(UD-col), y_(UD-row)) is maintained. As alluded to above, anddiscussed in more detail below, this value is an actual pixel locationmeasured in a gray scale image by the imaging analysis computer 130which may be corrected for distortion. A next image frame is to beacquired, in step 2505, such that between the two image frames, theimaging camera has moved a known distance, t_(diff), in the y directionand has not moved in either the x direction or z direction.

In the simplified, or undistorted, projection model, the perspectiveprojection equation (Equation 19A) provides that, in general, a worldpoint is projected to a location on an image plane according to Equation36A:

$\begin{bmatrix}u \\v \\\xi\end{bmatrix} = {{\lbrack K\rbrack\left\lbrack {R_{cam}❘T_{cam}} \right\rbrack}\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w} \\1\end{bmatrix}}$

where:

$\quad\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w} \\1\end{bmatrix}$are the homogenous coordinates for a world point located at (X_(W),Y_(W), Z_(w)) relative to a world coordinate system origin,

-   -   [K] is a 3×3 intrinsic camera matrix;    -   [R_(cam)] is a 3×3 rotation matrix describing the relative        orientation of a coordinate system of the camera and the world        coordinate system;    -   [T_(cam)] is 3×1 translation vector describing a displacement of        the camera origin location between two image frames;

$\quad\begin{bmatrix}u \\v \\\xi\end{bmatrix}$are the homogenous coordinates for a projected pixel location on theimage plane, which (referring back to Equation 19A) relate to pixellocations (x_(col), y_(row)) such that u=ξx_(col) and v=ξy_(row); and

-   -   ξ represents an unknown scale factor that can be estimated as a        state variable of a Kalman filter, as explained below.

Equation 36A can be used for two successive image frames in which thecamera moves between the acquisition of each frame. As between twosuccessive image frames, a translation vector,

${a\mspace{14mu}{translation}{\mspace{11mu}\;}{vector}},{= \begin{bmatrix}0 \\t_{diff} \\0\end{bmatrix}},$represents the relative translation of the camera between the two imageframes. In other words, if the camera position when the second imageframe is acquired is considered as being relative to the camera positionwhen the first image frame is acquired, then the translation vector forthe first image frame to be used in Equation 36A is

$T_{cam} = \begin{bmatrix}0 \\0 \\0\end{bmatrix}$and the translation vector for the second image frame to be used inEquation 36A is

$T_{cam} = {\begin{bmatrix}0 \\t_{diff} \\0\end{bmatrix}.}$

Thus, for the first image frame, Equation 36A provides an Equation 36B:

$\begin{bmatrix}u_{1} \\v_{1} \\\xi_{1}\end{bmatrix} = {{\lbrack K\rbrack\begin{bmatrix}r_{00} & r_{01} & r_{02} & 0 \\r_{10} & r_{11} & r_{12} & 0 \\r_{20} & r_{21} & r_{22} & 0\end{bmatrix}}\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w} \\1\end{bmatrix}}$This equation can be rearranged to provide Equation 36C:

${\lbrack K\rbrack^{- 1}\begin{bmatrix}u_{1} \\v_{1} \\\xi_{1}\end{bmatrix}} = {\begin{bmatrix}r_{00} & r_{01} & r_{02} & 0 \\r_{10} & r_{11} & r_{12} & 0 \\r_{20} & r_{21} & r_{22} & 0\end{bmatrix}\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w} \\1\end{bmatrix}}$

One property of any rotation matrix is that it is orthonormal whichmeans that

[R]^(T) [R]=I. Multiplying both sides by [R]^(T) and relabeling theleft-hand side with the following Equation 36D:

$\begin{bmatrix}a \\b \\c\end{bmatrix} = {\lbrack K\rbrack^{- 1}\begin{bmatrix}u_{1} \\v_{1} \\\xi_{1}\end{bmatrix}}$

allows Equation 36C to be transformed into Equation 36E:

${\left\lbrack R_{cam} \right\rbrack^{T}\begin{bmatrix}a \\b \\c\end{bmatrix}} = {{\left\lbrack R_{cam} \right\rbrack^{T}\begin{bmatrix}r_{00} & r_{01} & r_{02} & 0 \\r_{10} & r_{11} & r_{12} & 0 \\r_{20} & r_{21} & r_{22} & 0\end{bmatrix}}\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w} \\1\end{bmatrix}}$

which simplifies to Equation 36F:

${\left\lbrack R_{cam} \right\rbrack^{T}\begin{bmatrix}a \\b \\c\end{bmatrix}} = {\begin{bmatrix}1 & 0 & 0 & 0 \\0 & 1 & 0 & 0 \\0 & 0 & 1 & 0\end{bmatrix}\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w} \\1\end{bmatrix}}$

Because the location of the world point is in the form of homogeneouscoordinates (i.e.,

$\quad\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w} \\1\end{bmatrix}$), an equivalent set of homogeneous coordinates for this same worldpoint are also provided by:

$\begin{bmatrix}{\omega X}_{w} \\{\omega Y}_{w} \\{\omega Z}_{w} \\\omega\end{bmatrix}.$Using these equivalent homogeneous coordinates in Equation 36F providesfor Equation 36G:

${\left\lbrack R_{cam} \right\rbrack^{T}\begin{bmatrix}a \\b \\c\end{bmatrix}} = {\begin{bmatrix}1 & 0 & 0 & 0 \\0 & 1 & 0 & 0 \\0 & 0 & 1 & 0\end{bmatrix}\begin{bmatrix}{\omega X}_{w} \\{\omega Y}_{w} \\{\omega Z}_{w} \\\omega\end{bmatrix}}$

and multiplying out the right-hand side produces Equation 36H:

${\left\lbrack R_{cam} \right\rbrack^{T}\begin{bmatrix}a \\b \\c\end{bmatrix}} = \begin{bmatrix}{\omega X}_{w} \\{\omega Y}_{w} \\{\omega Z}_{w}\end{bmatrix}$

Using the homogenous coordinate representation of the world point thatalso includes ω, the projection equation for the second image frame isprovided by Equation 36I.

$\begin{bmatrix}u_{2} \\v_{2} \\\xi_{2}\end{bmatrix} = {{\lbrack K\rbrack\begin{bmatrix}r_{00} & r_{01} & r_{02} & 0 \\r_{10} & r_{11} & r_{12} & t_{diff} \\r_{20} & r_{21} & r_{22} & 0\end{bmatrix}}\begin{bmatrix}{\omega X}_{w} \\{\omega Y}_{w} \\{\omega Z}_{w} \\\omega\end{bmatrix}}$

where ω is another unknown scale factor that can be estimated as a statevariable of a Kalman filter as described below. Equation 36I incondensed notation can be written as Equation 36J:

$\begin{bmatrix}u_{2} \\v_{2} \\\xi_{2}\end{bmatrix} = {{\lbrack K\rbrack\left\lbrack R_{cam} \middle| T_{cam} \right\rbrack}\begin{bmatrix}{\omega X}_{w} \\{\omega Y}_{w} \\{\omega Z}_{w} \\\omega\end{bmatrix}}$ ${{where}\mspace{14mu} T_{cam}} = \begin{bmatrix}0 \\t_{diff} \\0\end{bmatrix}$

using Equation 36H on the right-hand side of Equation 36J providesEquation 36K:

$\begin{bmatrix}u_{2} \\v_{2} \\\xi_{2}\end{bmatrix} = {{\lbrack K\rbrack\left\lbrack R_{cam} \middle| T_{cam} \right\rbrack}\begin{bmatrix}{\left\lbrack R_{cam} \right\rbrack^{T}\begin{bmatrix}a \\b \\c\end{bmatrix}} \\\omega\end{bmatrix}}$

The right-hand side of Equation 36K, when multiplied out becomesEquation 36L below. This result relies on the rotation matrix beingorthonormal so that the rotation matrix multiplied by its transposeresults in an identity matrix.

$\begin{matrix}{\begin{bmatrix}u_{2} \\v_{2} \\\xi_{2}\end{bmatrix} = {\lbrack K\rbrack\left\lbrack {\begin{bmatrix}a \\b \\c\end{bmatrix} + {\omega\left\lbrack T_{cam} \right\rbrack}} \right\rbrack}} & {{Equation}\mspace{14mu} 36L}\end{matrix}$

Equation 36D can be substituted into Equation 36L to provide Equation36M:

$\begin{bmatrix}u_{2} \\v_{2} \\\xi_{2}\end{bmatrix} = {\lbrack K\rbrack\left\lbrack {{\lbrack K\rbrack^{- 1}\begin{bmatrix}u_{1} \\v_{1} \\\xi_{1}\end{bmatrix}} + {\omega\left\lbrack T_{cam} \right\rbrack}} \right\rbrack}$

This equation can be simplified to reveal Equation 36N:

$\begin{bmatrix}u_{2} \\v_{2} \\\xi_{2}\end{bmatrix} = {\begin{bmatrix}u_{1} \\v_{1} \\\xi_{1}\end{bmatrix} + {\lbrack K\rbrack\left\lbrack {\omega\left\lbrack T_{cam} \right\rbrack} \right\rbrack}}$

Thus, Equation 36N provides a model that describes how the projection ofa world point will change between successive image frames that areacquired such that the camera moves an amount t_(diff) in they-direction between the image frames. In particular, the model predictsthat the projection location

$\quad\begin{bmatrix}u_{2} \\v_{2} \\\xi_{2}\end{bmatrix}$for the next image frame can be calculated based on the projectionlocation

$\quad\begin{bmatrix}u_{1} \\v_{1} \\\xi_{1}\end{bmatrix}$of the previous image frame and the movement, T_(cam), of the imagingcamera 130 between the two image frames.

In particular, for

${\lbrack K\rbrack = \begin{bmatrix}f_{x} & \gamma & x_{0} \\0 & f_{y} & y_{0} \\0 & 0 & 1\end{bmatrix}},$Equation 36N providesu ₂ =u ₁ +ωγt _(diff) and v ₂ =v ₁ +ωf _(y) t _(diff)  Equation 37:

Thus, Equation 37 reveals that, in undistorted image space, a subsequentstate for a projection location can be calculated from the previousstate and a value t_(diff). The subsequent state relies on the valuesfor u₁, v₁, γ, ω, and f_(y) and also a current value for t_(diff).

This is the type of system or model with which a Kalman filter can beutilized to estimate the state of the system; in this particularexample, the state of the system can include an estimate of theprojected location of a scored candidate object being tracked betweenimage frames.

In general, a Kalman filter is a recursive algorithm that operates inreal time on a stream of noisy input observation data. In terms of theimage analysis system described herein, the observation data for ascored candidate object are the sequence of location coordinates of thatscored candidate object as it is identified and tracked through asequence of images. The Kalman filter also generates a prediction of afuture state of the system based on the model underlying the observationdata. In terms of the image analysis system described herein, theprediction is performed in step 2504 of FIG. 25 when an unadjustedpredicted location for a scored candidate object in a next image frameis calculated. The model's unadjusted predicted location, or preliminarypredicted location, is compared to the scored candidate object's actualmeasured image pixel location for that image frame and this differenceis scaled by a factor referred to as the “Kalman Gain”. The scaleddifference is provided as feedback into the model for the purpose ofadjusting the preliminary or unadjusted predicted location and therebyimproving subsequent predictions.

At each time step, or iteration, the Kalman filter produces estimates ofunknown values, along with their uncertainties. Once the outcome of thenext measurement is observed, these estimates are updated using aweighted average, with more weight given to estimates with loweruncertainty. The weights are calculated from the covariance, a measureof uncertainty, of the prediction of the system's state. The result is anew state estimate (or adjusted prediction) that lies in between theunadjusted or preliminary predicted system state and the measured, orobserved, system state. In many applications, such as this one, theinternal state of the modeled system can include more state variablesthan just the measurable or observable variables; however, the Kalmanfilter can be used to estimate all of the state variables. Thus,estimates for the variables γ, ω, and f_(y) can be calculated fordifferent iterations of the Kalman filter and their respective valuesmay vary over time such that respective values for γ(t), ω(t), and f_(y)(t) may be different for different iterations.

The Kalman filter begins with the following matrices: a state-transitionmodel Φ; an observation model H, a covariance of the modeled processnoise Q; and a covariance of the observation noise R. The covariancematrices Q and R can be experimentally calculated by making multipleobservations and analyzing the resulting data.

Using the conventional index “t” instead of the numerical subscripts ofEquation 37 to represent the iterative sequence of the differentvariables in the Kalman filter, Equation 37 provides two equations thatrepresent the model's current unadjusted prediction of the (û(t),{circumflex over (v)}(t) projection location of a scored candidateobject in undistorted image space in a next image frame based on itsprevious, estimated state or adjusted predicted projection location(u(t−1), v(t−1)) from a prior image frame where:û(t)=u(t−1)+(ω(t−1)γ(t−1)t _(diff)){circumflex over (v)}(t)=v(t−1)+(ω(t−1)f _(y)(t−1)t _(diff))

As explained more fully below, there are some variables in the Kalmanfilter for which a preliminary or unadjusted prediction is calculatedand then later adjusted. For a particular iteration of the Kalman filteran unadjusted predicted value for a variable is denoted with a ^ accentmark above the variable symbol. Based on the operation of the Kalmanfilter, however, this unadjusted predicted value can be adjusted with anupdated prediction value and a variable for an adjusted predicted valueis denoted by omitting the ^ accent mark above the variable symbol. Notall variables within the Kalman filter are first preliminarily predictedand then adjusted; those variables which are not adjusted may alsotypically be denoted by omitting the ^ accent mark above the variablename.

As discussed later, ξ, the scaling factor from the homogeneouscoordinates of previous equations (e.g., Equation 36A) is also includedas one of the state variables, or factors, that comprise the Kalmanfilter. Using these factors, one exemplary set of unadjusted predictionequations for a current unadjusted predicted state {circumflex over(Z)}(t) of a Kalman filter can be defined as:

$\left\lbrack {\hat{Z}(t)} \right\rbrack = \begin{bmatrix}{\hat{u}(t)} \\{\hat{v}(t)} \\{\hat{\xi}(t)} \\{\hat{\omega}(t)} \\{\hat{\gamma}(t)} \\{\hat{f_{y}}(t)}\end{bmatrix}$ where: û(t) = u(t − 1) + (ω(t − 1)γ(t − 1)t_(diff))v̂(t) = v(t − 1) + (ω(t − 1)f_(y)(t − 1)t_(diff)) ξ̂(t) = ξ(t − 1)ω̂(t) = ω(t − 1) γ̂(t) = γ(t − 1)${\hat{f_{y}}(t)} = {f_{y}\left( {t - 1} \right)}$

As for the initial unadjusted prediction, [{circumflex over (Z)}(1)], itcan be calculated using an initial state, Z(0), which can be initializedas:

${Z(0)} = \begin{bmatrix}{{u(0)} = {x_{OBS}(0)}} \\{{v(0)} = {y_{OBS}(0)}} \\{{\xi(0)} = 1} \\{{\omega(0)} = 0} \\{{\gamma(0)} = \gamma} \\{{f_{y}(0)} = f_{y}}\end{bmatrix}$

In the initial state, Z(0), the values for γ and f_(y) are from theintrinsic camera matrix [K] which may be calculated according to knowncamera resectioning techniques as noted above. The coordinates(x_(OBS)(0),y_(OBS) (0)) represent the new scored candidate object'smeasured or observed image location (in undistorted image space) in theinitial gray scale image frame in which that new scored candidate objectwas first identified by the image analysis computer 110.

From these equations, the state transition matrix, [Φ(t)] to be used inthe Kalman filter can be determined to be:

${\Phi(t)} = \begin{bmatrix}1 & 0 & 0 & {{\gamma\left( {t - 1} \right)}t_{diff}} & {{\omega\left( {t - 1} \right)}t_{diff}} & 0 \\0 & 1 & 0 & {{f_{y}\left( {t - 1} \right)}t_{diff}} & 0 & {{\omega\left( {t - 1} \right)}t_{diff}} \\0 & 0 & 1 & 0 & 0 & 0 \\0 & 0 & 0 & 1 & 0 & 0 \\0 & 0 & 0 & 0 & 1 & 0 \\0 & 0 & 0 & 0 & 0 & 1\end{bmatrix}$

Because of the non-linearity of the first two unadjusted predictionequations (i.e., (û(t) and {circumflex over (v)}(t)) with respect to theprevious estimated state, or adjusted predicted state, of the system(i.e., u(t−1) and v(t−1)), each calculation of the unadjusted predictedstate is estimated with a linearized model in the state transitionmatrix that takes the partial derivative of the state transitionequations with respect to the appropriate state variable. For example,to arrive at the first row of the above example state transition matrix,Φ(t), which corresponds to (û(t), the following partial derivatives canbe calculated for (û(t) to provide the coefficients in the first row ofthe state transition matrix Φ(t):

$\frac{\partial{\hat{u}(t)}}{\partial{u\left( {t - 1} \right)}} = 1$$\frac{\partial{\hat{u}(t)}}{\partial{v\left( {t - 1} \right)}} = 0$$\frac{\partial{\hat{u}(t)}}{\partial{\xi\left( {t - 1} \right)}} = 0$$\frac{\partial{\hat{u}(t)}}{\partial{\omega\left( {t - 1} \right)}} = {{\gamma\left( {t - 1} \right)}t_{diff}}$$\frac{\partial{\hat{u}(t)}}{\partial{\gamma\left( {t - 1} \right)}} = {{\omega\left( {t - 1} \right)}t_{diff}}$$\frac{\partial{\hat{u}(t)}}{\partial{f_{y}\left( {t - 1} \right)}} = 0$

The remaining elements of the state transition matrix [Φ(t)] arecalculated in a similar manner.

As mentioned briefly above, with Kalman filters, a current unadjustedpredicted state of a system {circumflex over (Z)}(t) is calculated froma previous estimated state of the system Z(t−1) according to the variousstate transition matrix coefficients. The notation {circumflex over(Z)}(t) denotes an unadjusted predicted state estimate of the systemthat has not been updated with a current observation or measurement atiteration “t”. The notation Z(t) denotes an adjusted predicted stateestimate that takes into consideration the current observation ormeasurement at iteration “t”.

As mentioned above, the current unadjusted predicted projection locationvalues are (û(t), {circumflex over (v)}(t) and these values representwhere a pallet object is predicted to be projected in undistorted imagespace in the next image frame. However, this current unadjustedpredicted projection location is in terms of homogenous coordinates andincludes a predicted scaling factor {circumflex over (ξ)}(t). To convertthe homogenous coordinates to current unadjusted predicted pixellocations ({circumflex over (x)}(t), ŷ(t)) in the image that areanalyzed by the image analysis computer 110, the scale factor is removedby division according to:

${\hat{x}(t)} = {{\frac{\hat{u}(t)}{\hat{\xi}(t)}\mspace{14mu}{and}\mspace{14mu}{\hat{y}(t)}} = \frac{\hat{v}(t)}{\hat{\xi}(t)}}$

This current unadjusted predicted image pixel location is in undistortedimage space and can be converted to distorted image space so that it canbe used as the center of the prediction circle for the next image frame(See step 2504 of FIG. 25). In particular, the current unadjustedpredicted image pixel location, mapped into distorted image space, isthe center of a prediction window for identifying potentially matchingnew scored candidate objects identified in a next image frame associatedwith the NewObjectlist. As subsequent frames are acquired, acorresponding current unadjusted predicted pixel location (in distortedpixel coordinates) is calculated and used as the center of theprediction window for each most recent next image frame.

For purposes of the Kalman filter, a “measurement” at iteration “t” isthe observed or measured location (i.e., image pixel coordinates) forthe new scored candidate object at iteration “t,” can be referred to as(x_(OBS)(t), y_(OBS)(t)) and is stored in the NewObjectslist. Thisobserved location is an image location in the next frame identified bythe image analysis computer 110 (measured in distorted image space andthen converted into an undistorted image location) to be the lowercenter point of the new scored candidate object in the next image frame(i.e., the image frame used in conjunction with the NewObjectlist).Because the Kalman filter predicted variables (û(t), {circumflex over(v)}(t)) are a predicted projection location using homogeneouscoordinates and the observed, or measured, variable is a pixel locationin the image, an observation matrix [H] is constructed that relates thepredicted variable (i.e., in homogeneous coordinates) with the measuredvariable (i.e., in pixel locations in the image frame). Thus, theobservation matrix is written in terms of ({circumflex over (x)}(t),ŷ(t)) where, from above:

${\hat{x}(t)} = {{\frac{\hat{u}(t)}{\hat{\xi}(t)}\mspace{14mu}{and}\mspace{14mu}{\hat{y}(t)}} = \frac{\hat{v}(t)}{\hat{\xi}(t)}}$

These two equations are non-linear with respect to the currentunadjusted predicted projection location (i.e., (û(t), {circumflex over(v)}(t))) and a linearized observation matrix is constructed, similar tothe state transition matrix above, by calculating partial derivatives ofeach equation with respect to the different state variables of theKalman filter.

In other words, the observation matrix is constructed according to:

${H(t)} = \begin{bmatrix}\frac{\partial{\hat{x}(t)}}{\partial{\hat{u}(t)}} & \frac{\partial{\hat{x}(t)}}{\partial{\hat{v}(t)}} & \frac{\partial{\hat{x}(t)}}{\partial{\hat{\xi}(t)}} & \frac{\partial{\hat{x}(t)}}{\partial{\hat{\omega}(t)}} & \frac{\partial{\hat{x}(t)}}{\partial{\hat{\gamma}(t)}} & \frac{\partial{\hat{x}(t)}}{\partial{\hat{f_{y}}(t)}} \\\frac{\partial{\hat{y}(t)}}{\partial{\hat{u}(t)}} & \frac{\partial{\hat{y}(t)}}{\partial{\hat{v}(t)}} & \frac{\partial{\hat{y}(t)}}{\partial{\hat{\xi}(t)}} & \frac{\partial{\hat{y}(t)}}{\partial{\hat{\omega}(t)}} & \frac{\partial{\hat{y}(t)}}{\partial{\hat{\gamma}(t)}} & \frac{\partial{\hat{y}(t)}}{\partial{\hat{f_{y}}(t)}}\end{bmatrix}$

These equations provide an observation, or measurement, matrix H(t) forthe Kalman filter where:

${H(t)} = \begin{bmatrix}\frac{1}{\hat{\xi}(t)} & 0 & {- \frac{\hat{u}(t)}{{\hat{\xi}}^{2}\;(t)}} & 0 & 0 & 0 \\0 & \frac{1}{\hat{\xi}(t)} & {- \frac{\hat{v}(t)}{{\hat{\xi}}^{2}\;(t)}} & 0 & 0 & 0\end{bmatrix}$

The residual of the Kalman filter, the difference between theobservation, or measurement, at iteration “t” (i.e., (x_(OBS)(t),y_(OBS)(t)),) and the scaled current unadjusted predicted estimate from{circumflex over (Z)}(t) (i.e., ({circumflex over (x)}(t), ŷ(t)) can becalculated as:

$\left\lbrack {ɛ_{r}(t)} \right\rbrack = {\begin{bmatrix}{x_{{OBS}\;}(t)} \\{y_{OBS}(t)}\end{bmatrix} - \begin{bmatrix}{\hat{x}(t)} \\{\hat{y}(t)}\end{bmatrix}}$

which can also be written as:

$\left\lbrack {ɛ_{r}(t)} \right\rbrack = {\begin{bmatrix}{x_{{OBS}\;}(t)} \\{y_{OBS}(t)}\end{bmatrix} - \begin{bmatrix}\frac{\hat{u}(t)}{\hat{\xi}(t)} \\\frac{\hat{v}(t)}{\hat{\xi}(t)}\end{bmatrix}}$

As known within Kalman filter techniques, an a priori predicted processcovariance matrix can be calculated according to:{circumflex over (P)}(t)=Φ(t)P(t−1)Φ(t)^(T) +Q

The residual covariance matrix can be calculated according to:S(t)=H(t){circumflex over (P)}(t)H(t)^(T) +R

The optimal Kalman Gain can be calculated according to:G(t)={circumflex over (P)}(t)H(t)^(T) S(t)⁻¹

The updated state estimate, Z(t), can then be calculated according to:Z(t)={circumflex over (Z)}(t)+G(t)[ε_(r)(t)]

This current adjusted predicted state estimate Z(t) will have an updatedprediction for the values (u(t), v(t)) which are considered to be theestimated values for these state variables to be used in a nextiteration of the Kalman filter. The updated state estimate Z(t) willalso have updated prediction values, or estimated values, for the otherfour state variables ω(t), γ(t), ξ(t), and f_(y)(t) to also be used inthe next iteration of the Kalman filter. In addition to storing theupdated Kalman filter state (i.e., Z(t)) in the ExistingObjectlist forthe new scored candidate object, the measured pixel location coordinatesin undistorted image space (i.e., (x_(OBS), y_(OBS))) may be added asthe next undistorted pixel location in the sequence of undistorted pixellocations stored in ExistingObjectlist for this particular scoredcandidate object. As described earlier, it is this sequence ofundistorted pixel locations in an ExistingObjectlist record for aparticular scored candidate object that are used in Equation 28A (orEquation 28B or Equation 31A) to calculate a pallet vector (X_(w),Y_(w), Z_(w)) for the scored candidate object.

The adjusted, a posteriori, process covariance matrix can be calculatedaccording to:P(t)=(I−G(t)H(t)){circumflex over (P)}(t)

As for initializing the Kalman filter, P(0) can be a 6×6 identitymatrix, Q can be a 6×6 matrix with every element=0, and R can be a 2×2diagonal matrix with each diagonal element=0.001.

FIG. 28 is a flowchart of an exemplary process highlighting furtherdetails of portions of the process of FIG. 25. As mentioned above, theKalman filter operates in undistorted image space with a current stateestimate, or updated predicted projection locations, u(t), v(t) from theprevious iteration of the Kalman filter.

For clarity, two particular image frames are selected to describe theoperation of the process of FIG. 28 with respect to a particular one ofthe possible scored candidate objects in the ExistingObjectlist. If, forexample, the prior image frame is n=4 and the next image frame is n=5,then the ExistingObjectlist will include the sequence of previouslydetermined corrected actual/measured image pixel locations, inundistorted image space, for this scored candidate object. For example,the most recent corrected actual image pixel location for this scoredcandidate object in the ExistingObjectlist has coordinates (x_(UD-col) ₄, y_(UD-row) ₄ ). In addition, as shown in step 2802, theExistingObjectlist will also store the current, adjusted predicted statevariable values as calculated by the immediately previous iteration ofthe Kalman filter. In other words, for image frame n=4, theExistingObjectlist stores (in terms of the Kalman filter variables, acurrent estimated state of the system state variables (u(t−1), v(t−1)),wherein the location values (u(t−1), v(t−1)) are defined by the updatedpredicted projection location values (u(t), v(t)) from the previousiteration of the Kalman filter.

In step 2804, the image analysis computer 110 implementing the Kalmanfilter, may use this current estimated state (u(t−1), v(t−1)) and theamount of camera movement t_(diff) between image frames to calculate acurrent unadjusted predicted estimate {circumflex over (Z)}(t). Thisprediction represents what the underlying physical model predicts(without any type of adjustment) the projection location (e.g., (û(t),{circumflex over (v)}(t))) of the world point (X_(W), Y_(W), Z_(W)) inthe next image frame (i.e., image frame n=5) will be because of thecamera movement t_(dff) that occurred between the two image frames.

The homogeneous coordinate values of the unadjusted predicted projectionlocation (û(t), {circumflex over (v)}(t)) are scaled, in step 2806, bydividing by {circumflex over (ξ)}(t) to determine a correspondingunadjusted predicted pixel location ({circumflex over (x)}(t), ŷ(t)) inthe next image frame (i.e., frame n=5). This unadjusted predicted pixellocation is in undistorted image space, and is converted, in step 2808,to distorted image pixel coordinates. These distorted image pixelcoordinates are considered the scored candidate object's unadjustedpredicted new pixel location (in distorted image space) in the nextimage frame n=5, as shown in step 2504 of FIG. 25. In step 2810, theunadjusted predicted new location and the prediction window are definedin image frame n=5 with reference to these distorted pixel coordinates.

Assuming that in step 2512 of FIG. 25, that a new scored candidateobject in image frame n=5 is identified, located, and matched with thisparticular existing candidate object from the ExistingObjectlist, thenthat new scored candidate object will have an observed, or measured,image pixel location in distorted image space in image frame n=5 of(x _(D-col) ₅ ,y _(D-row) ₅ )

The image analysis computer 110, in step 2812, considers this imagepixel location as the uncorrected measurement location for this scoredcandidate object to be used in the current iteration of the Kalmanfilter.

In step 2814, the image analysis computer corrects, or converts, thedistorted image pixel location coordinates (x_(D-col) ₅ , y_(D-row) ₅ )into measured undistorted image location coordinates (x_(UD-col) ₅ ,y_(UD-row) ₅ ). These measured undistorted image location coordinates(x_(UD-col) ₅ , y_(UD-row) ₅ ) are considered as the current measurement(x_(OBS)(t), y_(OBS)(t)), and are compared to the unadjusted predictedpixel location ({circumflex over (x)}(t), ŷ(t)) for frame n=5 by theKalman filter to calculate the residual, [ε_(r)(t)], and to update thestate of the filter, as shown in step 2524 of FIG. 25.

Once the state of the Kalman filter is updated, the adjusted prediction,or estimated state, Z(t) will have adjusted prediction values for (u(t),v(t)) as well as for the other state variables for the new scoredcandidate object. There will also be an adjusted process covariancematrix P(t). These adjusted prediction values for (u(t), v(t)) are whatthe Kalman filter indicates are the current estimated values for thesestate variables that are to be used in the preliminary, or unadjusted,prediction during the next iteration of the Kalman filter. Thus, in step2816, the image analysis computer 110 stores this adjusted prediction,or estimated state, Z(t) for this scored candidate object in theExistingObjectlist.

In addition to storing the updated Kalman filter state in theExistingObjectlist for the scored candidate object, the measured pixellocation coordinates in undistorted image space (i.e., (x_(OBS),y_(OBS))) may then be added, in step 2818, as the next undistorted pixellocation in the sequence of undistorted pixel locations stored inExistingObject list for this particular scored candidate object.

It is these undistorted image pixel coordinates (x_(OBS), y_(OBS)) thatare stored as the corrected actual undistorted coordinates (x_(UD-col) ₅, y_(UD-row) ₅ ) for the object's location in image frame n=5. Asmentioned above, it is the sequence of these undistorted coordinatesthat are used by other processes to periodically calculate a palletvector (X_(W), Y_(W), Z_(W)) for the scored candidate object.

Application of the Kalman filter is repeatedly applied in a similarmanner as this scored candidate object is tracked through subsequentimage frames n=6, 7, 8, . . . .

The above description of acquiring images, tracking objects, andestimating a pallet vector assumes that the vehicle 20 is stationarywhile the carriage apparatus 40 with the imaging camera 130 travels upand down in the Y direction. However, even if the vehicle 20 is movingduring a lift operation, this movement can be accounted for.

In particular, the vehicle 20 can move in the X direction (i.e., lateralto a pallet face) and the Z direction (i.e., towards or away from apallet face) but it is presumed that the vehicle 20 does not move in theY direction (i.e., float up or down). Also, the vehicle 20 is presumednot to rotate about the X axis (i.e., tilt forwards or back), presumednot to rotate about the Z axis (i.e., tilt side-to-side), and presumedto be able to rotate about the Y axis (i.e., adjust how square thevehicle 20 is relative to the pallet face).

Under the previous assumption of no vehicle movement, the origin of thephysical world coordinate system was considered a constant. Thus, therewere a series of image point pallet locations (x_(col) _(n) , y_(row)_(n) ) which were different observations of the same physical worldpallet location (X_(w), Y_(w), Z_(w)) and the index n referred to theimage frame number. However, under the presumption that the vehicle maymove between image frames, the location in physical world coordinates(X_(w) _(n) , Y_(w) _(n) , Z_(w) _(n) ) now also includes the indexvalue n because the coordinates in the physical world are relative to aworld coordinate system origin, O_(n), at a time when image frame n isacquired. For example, the world coordinate system origin, O_(n) may belocated on the frame of the vehicle 20. If between image frame n andimage frame n+1 the vehicle 20 moves, then this origin, O_(n), moves andthe location 2704 of the pallet 2702 in the physical world will havecoordinates (X_(w) _(n+1) , Y_(w) _(n+1) , Z_(w) _(n+1) ) relative tothe world coordinate system origin, O_(n+1), at a time that the imageframe n+1 is acquired.

The concepts relating to transformation between two different coordinatesystems have been described above with respect to Equations 15 and 21,which, applying these techniques in the context of movement of the truck20 between images, provide

$\begin{matrix}{\begin{bmatrix}X_{w_{n}} \\Y_{w_{n}} \\Z_{w_{n}}\end{bmatrix} = {\begin{bmatrix}\; & {tt}_{x_{n}} \\\left\lbrack R_{truck} \right\rbrack_{3 \times 3} & {tt}_{y_{n}} \\\; & {tt}_{z_{n}}\end{bmatrix}\begin{bmatrix}X_{w_{({n - 1})}} \\Y_{w_{({n - 1})}} \\Z_{w_{({n - 1})}} \\1\end{bmatrix}}} & {{Equation}\mspace{14mu} 38A}\end{matrix}$which in homogenous coordinates is

$\begin{matrix}{\begin{bmatrix}X_{w_{n}} \\Y_{w_{n}} \\Z_{w_{n}} \\1\end{bmatrix} = {\begin{bmatrix}\; & \; & \; & {tt}_{x_{n}} \\\; & \left\lbrack R_{{truck}_{n}} \right\rbrack_{3 \times 3} & \; & {tt}_{y_{n}} \\\; & \; & \; & {tt}_{z_{n}} \\0 & 0 & 0 & 1\end{bmatrix}\begin{bmatrix}X_{w_{({n - 1})}} \\Y_{w_{({n - 1})}} \\Z_{w_{({n - 1})}} \\1\end{bmatrix}}} & {{Equation}\mspace{14mu} 38B}\end{matrix}$

where:

-   -   (X_(w) _(n) , Y_(w) _(n) , Z_(w) _(n) ) are the physical world        coordinates of the scored candidate object relative to the world        coordinate system origin location, O_(n) when image frame n is        acquired;    -   (X_(w) _((n−1)) , Y_(w) _((n−1)) , Z_(w) _((n−1)) ) are the        physical world coordinates of the scored candidate object        relative to the world coordinate system origin location,        O_(n−1), when image frame n−1 is acquired;    -   [R_(truck) _(n) ]_(3×3) is the rotation matrix based on the        rotation of the respective world origins between image frame n−1        and image frame n (i.e., the rotation of the vehicle 20 around        the Y axis, θ_(y) _(n) ); and

$\quad\begin{bmatrix}{tt}_{x_{n}} \\{tt}_{y_{n}} \\{tt}_{z_{n}}\end{bmatrix}$is the translation matrix based on the movement of the respective worldorigins between image frame n−1 and image frame n (i.e., the movement inthe position of the vehicle 20 between the two image frames).

The amount that the vehicle 20 rotates or moves between frames can bemeasured, or calculated, by various sensors in communication with thevehicle computer 50. For example, a steering angle sensor can be coupledwith a steering control mechanism of the vehicle 20 to provide the angleθ_(y) _(n) used in the rotation matrix of Equation 38B. Additionally,odometry data from wheel rotation sensors, or similar sensors and datafrom the steering angle sensor, can be used to calculate thetranslational movement of the vehicle 20, i.e., can be used to determinemovement along the X and Z directions. The vehicle computer 50 can thencommunicate with the image analysis computer 110 to provide theappropriate information.

Equation 38B can be written as Equation 38C:

$\begin{bmatrix}X_{w_{n}} \\Y_{w_{n}} \\Z_{w_{n}} \\1\end{bmatrix} = {\left\lbrack F_{{truck}_{n}} \right\rbrack\begin{bmatrix}X_{w_{({n - 1})}} \\Y_{w_{({n - 1})}} \\Z_{w_{({n - 1})}} \\1\end{bmatrix}}$

where, [F_(truck) _(n) ], given the presumptions about how the vehicle20 can move between image frames, is:

$\left\lbrack F_{{truck}_{n}} \right\rbrack = \begin{bmatrix}{\cos\;\theta_{y_{n}}} & 0 & {\sin\;\theta_{y_{n}}} & {tt}_{x_{n}} \\0 & 1 & 0 & 0 \\{{- \sin}\;\theta_{y_{n}}} & 0 & {\cos\;\theta_{y_{n}}} & {tt}_{z_{n}} \\0 & 0 & 0 & 1\end{bmatrix}$Rearranging Equation 38C can be accomplished to provide

$\begin{matrix}{{\left\lbrack F_{{truck}_{n}}^{- 1} \right\rbrack\begin{bmatrix}X_{w_{n}} \\Y_{w_{n}} \\Z_{w_{n}} \\1\end{bmatrix}} = \begin{bmatrix}X_{w_{({n - 1})}} \\Y_{w_{({n - 1})}} \\Z_{w_{({n - 1})}} \\1\end{bmatrix}} & {{Equation}\mspace{14mu} 38D}\end{matrix}$wherein the noted inverse matrix is given by:

$\left\lbrack F_{{truck}_{n}}^{- 1} \right\rbrack = \begin{bmatrix}{\cos\;\theta_{y_{n}}} & 0 & {{- \sin}\;\theta_{y_{n}}} & {{\sin\;\theta_{y_{n}}{tt}_{z_{n}}} + \frac{\left( {{\sin\;\theta_{y_{n}}} - 1} \right){tt}_{x_{n}}}{\cos\;\theta_{y_{n}}}} \\0 & 1 & 0 & 0 \\{\sin\;\theta_{y_{n}}} & 0 & {\cos\;\theta_{y_{n}}} & {- \left( {{\cos\;\theta_{y_{n}}{tt}_{z_{n}}} + {\sin\;\theta_{y_{n}}{tt}_{x_{n}}}} \right)} \\0 & 0 & 0 & 1\end{bmatrix}$

Under the presumption that the vehicle 20 may now move between imageframes, Equation 19A can be augmented by adding appropriate subscriptsto the different variables to provide Equation 39A:

${\lambda\begin{bmatrix}x_{{col}_{({n - 1})}} \\y_{{row}_{({n - 1})}} \\1\end{bmatrix}} = {{\lbrack K\rbrack\left\lbrack R_{cam} \middle| T_{{cam}_{({n - 1})}} \right\rbrack}\begin{bmatrix}X_{w_{({n - 1})}} \\Y_{w_{({n - 1})}} \\Z_{w_{({n - 1})}} \\1\end{bmatrix}}$

Equation 39A relates a physical world location (X_(w) _((n−1)) , Y_(w)_((n−1)) , Z_(w) _((n−1)) ), up to a scale factor λ, with a gray scaleimage pixel location (x_(col) _((n−1)) , y_(row) _((n−1)) ).Substitution of the left side of Equation 38D into the right side ofEquation 39A will yield Equation 39B:

${\lambda\begin{bmatrix}x_{{col}_{({n - 1})}} \\y_{{row}_{({n - 1})}} \\1\end{bmatrix}} = {{{\lbrack K\rbrack\left\lbrack R_{cam} \middle| T_{{cam}_{({n - 1})}} \right\rbrack}\left\lbrack F_{{truck}_{n}}^{- 1} \right\rbrack}\begin{bmatrix}X_{w_{n}} \\Y_{w_{n}} \\Z_{w_{n}} \\1\end{bmatrix}}$

which provides the previous image location point (x_(col) _((n−1)) ,y_(row) _((n−1)) ) in terms of a physical world location (X_(w) _(n) ,Y_(w) _(n) , Z_(w) _(n) ) relative to a new world coordinate systemorigin, O_(n), at the time the image frame n is acquired. An equivalentEquation 39B can be written for any of the previous image locationpoints. For example, Equation 39B can be written as:

$\begin{matrix}{{\lambda\begin{bmatrix}x_{{col}_{({n - 2})}} \\y_{{row}_{({n - 2})}} \\1\end{bmatrix}} = {{{\lbrack K\rbrack\left\lbrack R_{cam} \middle| T_{{cam}_{({n - 2})}} \right\rbrack}\left\lbrack F_{{truck}_{({n - 1})}}^{- 1} \right\rbrack}\begin{bmatrix}X_{w_{({n - 1})}} \\Y_{w_{({n - 1})}} \\Z_{w_{({n - 1})}} \\1\end{bmatrix}}} & {{Equation}\mspace{14mu} 39C}\end{matrix}$

and substituting Equation 38D into Equation 39C provides Equation 39D:

${\lambda\begin{bmatrix}x_{{col}_{({n - 2})}} \\y_{{row}_{({n - 2})}} \\1\end{bmatrix}} = {{{{\lbrack K\rbrack\left\lbrack R_{cam} \middle| T_{{cam}_{({n - 2})}} \right\rbrack}\left\lbrack F_{{truck}_{({n - 1})}}^{- 1} \right\rbrack}\left\lbrack F_{{truck}_{n}}^{- 1} \right\rbrack}\begin{bmatrix}X_{w_{n}} \\Y_{w_{n}} \\Z_{w_{n}} \\1\end{bmatrix}}$

Thus, Equations 39C and 39D can be generalized as Equation 39E:

${\lambda\begin{bmatrix}x_{{col}_{({n - N})}} \\y_{{row}_{({n - N})}} \\1\end{bmatrix}} = {{\lbrack K\rbrack\left\lbrack R_{cam} \middle| T_{{cam}_{({n - N})}} \right\rbrack}{\left( {\prod\limits^{k = {N\mspace{11mu}\ldots\mspace{11mu} 1}}\;\left\lbrack F_{{truck}_{({n - k + 1})}}^{- 1} \right\rbrack} \right)\begin{bmatrix}X_{w_{n}} \\Y_{w_{n}} \\Z_{w_{n}} \\1\end{bmatrix}}}$

This Equation allows any previous image location coordinates (x_(col)_((n−N)) , y_(row) _((n−N)) ) from image frame (n−N) to be expressed interms of a physical world location (X_(w) _(n) , Y_(w) _(n) , Z_(w) _(n)) relative to the new world coordinate system origin, O_(n), at the timethe image frame n is acquired. “N” corresponds to a desired previousimage frame and is equal to the number of frames it is located prior tocurrent frame “n.” In Equation 39E, the multiplication series isperformed from k=N to k=1.

Equation 39E is similar in structure to Equation 19A, which from above:

$\begin{matrix}{{\lambda\begin{bmatrix}x_{col} \\y_{row} \\1\end{bmatrix}} = {\lbrack K\rbrack\left\lbrack {R_{cam}{\left. T_{cam} \right\rbrack\begin{bmatrix}X_{w} \\Y_{w} \\Z_{w} \\1\end{bmatrix}}} \right.}} & {{Equation}\mspace{14mu} 19A}\end{matrix}$

Thus, a cross-product equation similar to Equation 20 can be written asEquation 40:

$\begin{bmatrix}x_{{col}_{({n - N})}} \\y_{{row}_{({n - N})}} \\1\end{bmatrix} \times {\lbrack K\rbrack\left\lbrack {{R_{cam}\left. T_{{cam}_{({n - N})}} \right\rbrack{\left( {\prod\limits^{k = {{N\ldots}\; 1}}\;\left\lbrack F_{{truck}_{({n - k + 1})}}^{- 1} \right\rbrack} \right)\begin{bmatrix}X_{w_{n}} \\Y_{w_{n}} \\Z_{w_{n}} \\1\end{bmatrix}}} = 0} \right.}$

In Equation 20A, the extrinsic camera matrix [R_(cam)|T_(cam)] is a 3×4matrix. In Equation 40, a similar but more complex 3×4 extrinsic cameramatrix

$\left\lbrack {R_{cam}\left. T_{{cam}_{({n - N})}} \right\rbrack\left( {\prod\limits^{k = {{N\ldots}\; 1}}\;\left\lbrack F_{{truck}_{({n - k + 1})}}^{- 1} \right\rbrack} \right)} \right.$is present as well, but Equation 40 is conceptually the same as Equation20A. Thus, similar to how Equation 20A was manipulated to produce twolinear equations in Equation 23, Equation 40 can also be manipulated toproduce two similar, but more complex, linear equations as well. Inparticular, the two linear equations will provide equations that relatean image pixel location, (x_(col) _((n−N)) , y_(row) _((n−N)) ), for animage frame n−N, to physical world location coordinates (X_(w) _(n) ,Y_(w) _(n) , Z_(w) _(n) ) of a physical object relative to the worldcoordinate system origin, O_(n), when the image frame n is acquired.This is true even though that same physical object had differentphysical world location coordinates (X_(w) _((n−N)) , Y_(w) _((n−N)) ,Z_(w) _((n−N)) ) relative to the world coordinate system origin,O_((n−N)), when the image frame (n−N) is acquired. As described before,with respect to Equation 25A and Equation 26A, these pairs of linearequations can be collected, respectively, for multiple observations ofthe physical object so that a pallet vector (X_(w) _(n) , Y_(w) _(n) ,Z_(w) _(n) ) can be calculated that may represent, for example, thelocation of the lower center point of a physical pallet relative to theworld coordinate system origin, O_(n), when the image frame n isacquired. As discussed above, an encoder may be provided to generatepulses to the vehicle computer 50 in response to movement of the forkcarriage apparatus 40 relative to the third weldment 36. The worldcoordinate origin, O_(n), when the image frame n is acquired, is fixedrelative to the third weldment so as to allow the vehicle computer 50 todetermine the linear displacement of the imaging camera 130 and theforks 42A and 42B relative to the world coordinate origin, O_(n) whenthe image frame n is acquired. Hence, the location of the imaging camera130 and the forks 42A and 42B relative to the world coordinate origin,O_(n) when the image frame n is acquired, can be provided by the vehiclecomputer 50 to the image analysis computer 110. This pallet vector canthen be used by vehicle computer 50 to guide the forks 42A and 42B toengage the physical pallet.

As described above, the image analysis computer 110 identifies andtracks a number of scored candidate objects through a series of images.A respective record for each scored candidate object is maintained inthe ExistingObjectlist. A corresponding record for a particular scoredcandidate object can include, among other values, the current position(X_(W), Y_(W), Z_(W)) of the scored candidate object as well as thecomposite object score, Score_(Object), for the scored candidate object.The corresponding record for a scored candidate object can also includea value Tag_(Object) that, as discussed below, can be associated with acutout, but is more generally associated with a thread of matchedobjects, where the cutout is defined as the height where the forks 42Aand 42B are to stop at a pallet. Cutouts themselves have a tag,Tag_(cutout). Cutouts are updated whenever certain conditions, describedbelow, of a scored candidate object are met or when Tag_(Object) equalsTag_(cutout).

The Y_(W) coordinate can be used to refer to the scored candidateobject's height in the physical world where the vehicle 20 operates.Using this information for all of the scored candidate objects, theimage analysis computer can calculate cutout location information,discussed below, that can be communicated to the vehicle computer 50 foruse in controlling operations such as, for example, fork placement andmovement. For example, once a scored candidate object has beenidentified in two different image frames that were respectively acquiredby the imaging camera 130 when the vehicle forks 42A and 42B were atdifferent heights, a pallet vector, or current position, (X_(W), Y_(W),Z_(W)) of the scored candidate object relative to the world coordinateorigin 2706 can be estimated. As discussed above, an encoder may beprovided to generate pulses to the vehicle computer 50 in response tomovement of the fork carriage apparatus 40 relative to the thirdweldment 36, which is fixed relative to the world coordinate origin2706, so as to allow the vehicle computer 50 to determine the lineardisplacement along the length of the mast assembly 30 of the forks 42Aand 42B and the imaging camera 130. Hence, the location of the forks 42Aand 42B relative to the world coordinate origin 2706 can be provided bythe vehicle computer 50 to the image analysis computer 110. Because thevehicle computer 50 provides the image analysis computer 110 with thevalue t_(y), the imaging analysis computer 110 can determine thelocation of the imaging camera 130 relative to the world coordinateorigin 2706 by summing the values of t_(y) and t_(ybias). As describedbelow, the pallet vectors for the scored candidate objects within acurrent field of view of the imaging camera 130 can be used, by theimage analysis computer 110, to calculate “cutouts,” i.e., a height offork receiving openings 210 and 212 within a pallet relative to theworld coordinate origin 2706, at which the forks 42A and 42B are tostop. The image analysis computer 110 can then periodically send thiscutout location information to the vehicle computer 50. For example, thecutout location information can be communicated every time it is updatedor a message with the cutout information can be communicated on apredetermined periodic schedule.

As noted above, in operation, an operator raises the forks 42A and 42Bvertically in the Y-direction via actuation of the multifunctioncontroller 14 to a position above the last pallet P which is to beignored. Then, while the forks 42A and 42B are still moving, theoperator actuates the trigger switch 140. Independent of the actuationof the trigger switch 140, the image analysis computer 110 causes theimaging camera 130 to take image frames, such as at a rate of 10-30 fps(frames/second), as the fork carriage apparatus 40 continues to movevertically. The image analysis computer 110 analyzes the images,identifies one or more objects in the image frames, determines which ofthe one or more objects most likely comprises a pallet, tracks the oneor more objects in a plurality of image frames, determines the locationsof the one or more objects relative to the world coordinate origin 2706and generates and transmits object location information, in the form ofcutout location values, to the vehicle computer 50. After receiving suchcutout location values, the vehicle computer 50 controls the forkmovement such that it automatically stops the forks, once the triggerswitch 140 is actuated, at a new first current stop height or cutoutlocation value.

It is noted that one or more heights or cutout location values at whichthe forks can be automatically stopped can be calculated by the imageanalysis computer 110. For example, a new first stop height can beassociated with a scored candidate object that has a vertical heightthat is closest to the present fork height of the vehicle 20. Also,there may be additional stop heights calculated that respectivelycorrespond to scored candidate objects above the new first stop height.

As mentioned, these stop heights can be conveniently referred to as“cutouts” or “cutout location values” and the image analysis computer110 may, for example, calculate three different cutouts (or more orless) that respectively correspond to a first scored candidate objectnear the current fork position, a second scored candidate object abovethe first scored candidate object, and a third scored candidate objectabove the second scored candidate object. In particular, as described inmore detail below, a cutout can be set equal to, or otherwise calculatedfrom, the current Y_(W) value that is associated with a scored candidateobject from its record in the ExistingObjectlist. Accordingly, eachcutout is inherently associated with a particular scored candidateobject that was used to set that particular cutout value. As mentionedabove, each scored candidate object also has a tag value, Tag_(Object),in its record in the ExistingObjectlist. Therefore, in addition tomerely maintaining one or more cutout values, each cutout value also hasa corresponding tag value, Tag_(Cutout), that is equal to theTag_(Object) value of the scored candidate object that was used to setthe cutout value.

FIGS. 29A-29F depict flowcharts of an exemplary method for calculatingcutouts in accordance with the principles of the present invention. Asmentioned above, one or more of the cutout location values may then beused by the vehicle computer 50 along with the trigger switch 140 tocontrol movement of the forks 42A and 42B. In particular, operatoractuation of the trigger switch 140 while the forks 42A and 42B arebeing raised will cause the vehicle computer 50 to, first, slow thespeed at which the forks 42A and 42B are rising (e.g., to about 14inches/second) and, secondly, cause the forks 42A and 42B to stop at anew first stop height or cutout location value that was provided by theimage analysis computer 110, as will be discussed below with regards tostep 2982. As explained below with respect to the method of FIGS.29A-29F, this new first cutout location value is an adjusted cutoutvalue calculated by the imaging analysis computer 110 to be closest to,and greater than, a current height of the forks 42A and 42B.

Initially, in step 2902, the image analysis computer 110, as part of itsinitialization steps to begin image acquisition, object locating, andobject tracking, also initializes all cutouts to equal zero. Secondly,there is an adjusted cutout value, described below, that the imageanalysis computer uses to adjust the first cutout value to a new firstcutout location value and this is initialized to zero as well. Theadjusted cutout value defining the new first cutout location value isequal to an estimated height of the first cutout value plus 1.5 inchesin the illustrated embodiment relative to the world coordinate originand is based on the Y_(w) coordinate of a pallet vector (X_(w), Y_(w),Z_(w)) of a corresponding scored candidate object whose values are alsorelative to the world coordinate origin. (See, for example, palletlocation 2704 of FIG. 27A. The additional amount of 1.5 inches in theillustrated embodiment corresponds to 3 inch high fork receivingopenings 210 and 212 and is added to the cutout location value so as toensure that the forks 42A and 42B enter the fork receiving openings 210and 212 and don't contact the bottom pallet board 200. Also, a counteris initialized wherein the counter has a value that reflects acertainty, or confidence, of whether an adjustment should be made to thefirst cutout value to arrive at an adjusted cutout value that iscommunicated to the vehicle computer 50. Additionally, other values usedthroughout the process of FIGS. 29A-29F are initialized as well in step2902. As noted above, in operation, the vehicle computer 50 moves thetruck forks based on the control input by an operator via themultifunction controller 14. This fork movement is depicted in theflowchart as step 2904 and at a given moment in time the forks are at aparticular “fork height”. The vehicle computer 50 can communicate thisinformation to the image analysis computer 110 to allow the imageanalysis computer 110 to perform the cutout calculations describedbelow.

As noted above, initially, all the cutout values equal zero. However, asdescribed below, a cutout value is eventually set equal or nearly equalto the height (e.g., the Y_(W) coordinate) of a particular scoredcandidate object. As mentioned, the image analysis computer 110 canmaintain a plurality of cutouts. In the example flowcharts of FIGS.29A-29F, three cutouts are maintained. The first cutout relates to acurrent cutout height or current cutout location value and the secondand third cutouts relate to cutout heights above the first cutout.

Once the image analysis computer 110 has pallet vectors available forone or more scored candidate objects, the computer 110 knows thedistance in the Y direction for those one or more scored candidateobjects relative to the world coordinate origin. Once the computer 110knows the height of the one or more scored candidate objects relative tothe world coordinate origin, the subsequent steps of the process ofFIGS. 29A-29F will set one or more of the three cutout values based onthose various scored candidate objects. Once the forks 42A and 42B ofthe vehicle are raised above a current first cutout location value(e.g., cutout[0]), the current first cutout value is no longer neededand the second cutout location value can become the first cutout value.In turn, the second cutout value can be set to equal the third cutoutlocation value and the third cutout value is initialized to be equal tozero. As described later (in step 2946), the third cutout can be set toa new value under certain circumstances as well. In this way, the imageanalysis computer 110 may maintain three cutout values but they can beupdated to ensure that all three cutout values are above the currentheight of the forks 42A and 42B. Because there can be some uncertaintyin a scored object's actual height (e.g., its Y_(W) coordinate), theimage analysis computer 110 can check to see if the first cutout valueis less than the current fork height minus a small amount, e.g., (“forkheight”−1.5 inches (the “put-to-center value”)) rather than determiningif the first cutout value is less than precisely the current “forkheight”. Based on this determination, in step 2906, the image analysiscomputer 110 can shift the cutout values, in step 2908, as describedabove or proceed directly to step 2910. The counter that was initializedin step 2902 was associated with a confidence in the current firstcutout value. Thus, once the current first cutout value is discarded andreplaced with the current second cutout value in step 2908, this counteris re-initialized as well. Also, in step 2908, the corresponding tagvalues (e.g., Tag_(cutout)[k]) of the different cutouts can be shiftedleft similar to the way the cutout values were shifted.

The activity occurring in step 2910 can occur outside of the logicalflow of the process depicted in FIGS. 29A-29F. However, it is depictedin FIG. 29B to indicate that the activity of step 2910 occurs so thatthe results of this activity can be used in the later steps of theflowcharts. In particular, in step 2910, the imaging analysis computer110 ensures that as each new scored candidate objects is first added tothe ExistingObjectlist, the record for that object includes a respectivevalue for Tag_(Object).

As shown in step 2902, a value next_Tag is initialized equal to one.When the very first scored candidate object is added to theExistingObjectlist, its corresponding Tag_(Object) value is set equal tonext_Tag (i.e., set equal to “1”), in step 2910. The value for next_Tagis also incremented in step 2910 so that the next newly identifiedscored candidate object (i.e., see step 2384 of FIG. 23E) that is addedto ExistingObjectlist has its Tag_(Object) value set equal to next_Tag(i.e., set equal to “2”) by step 2910. Thus, the activity of step 2910represents how each newly identified object that is added toExistingObjectlist is assigned a unique “tag” value based on a currentvalue of next_Tag.

In step 2912, the image analysis computer 110 identifies n scoredcandidate objects in the ExistingObjectlist which meet certain criteriato be cutout candidates. For example, those scored candidate objectswhich have a) either a composite object score, Score_(Object), greaterthan a predetermined threshold (e.g., 0.5), or a tag value, Tag_(Object)equaling Tag_(cutout)[0], the tag currently associated with the firstcutout, cutout[0] and b) a Y_(W) coordinate that is greater than apredetermined minimum height (e.g., 35 inches) and greater than thecurrent “fork height” (e.g., (fork height−1.5 inches), and a Z_(W)coordinate greater than zero can be identified as possible scoredcandidate objects to use as cutout candidates for setting cutout values.If no such cutout candidates are identified, the image analysis computercan continue with step 2968.

As discussed with relation to FIG. 24B, in the ExistingObjectlist, theimage analysis computer 110 can uniquely refer to each scored candidateobject by a respective index value and, thus, at the conclusion of step2912 a list of these index values can be identified that reference thosen scored candidate objects in the ExistingObjectlist that remain aspossible candidates to use for setting cutout values. As shown in step2916, the image analysis computer 110 can perform a number of steps foreach of the remaining scored candidate objects with index values in thelist. Once all of these scored candidate objects, or cutout candidates,have been utilized, this portion of the process is finished and, in step2966, the image analysis computer continues with step 2970. As describedlater with respect to step 2922, there are instances in which it isbeneficial to determine if there are cutout candidates within apredetermined distance of each of the cutouts. Thus, in step 2914, avalue cutoutFound[k] is initialized before the loop of step 2916 begins.As described below, the actual value for cutoutFound[k] will reflect ifstep 2922 was satisfied for cutout[k].

In step 2918, for a particular one of the remaining scored candidateobjects, or cutout candidates, the image analysis computer 110 loopsthrough and performs a number of steps for each of the three cutouts. Instep 2920, the image analysis computer 110 determines if the currentcutout value has been set, or in other words, is the current cutoutvalue not equal to zero. If the current cutout value has been set, thenthe image analysis computer continues with step 2922. If the currentcutout value has not been set, then the image analysis computercontinues with step 2938.

In step 2922, the image analysis computer 100 determines if the currentcutout value is within a predetermined height difference as compared toY_(W) of the scored candidate object of the current iteration of theloop initiated in step 2916. The cutouts can generically be referred toas an array cutout[j] where j=0, 1 or 2 (in the example where there arethree cutouts). Thus in step 2922, the image analysis computer candetermine if |cutout[j]−Y_(W)|<15 inches. If so, then the image analysiscomputer can perform steps 2924, 2926 and 2928. If not, then the imageanalysis computer 110 continues with step 2950 to move to the nextcutout value (i.e., cutout[j+1]) if available. Typically, only onecutout will be within 15 inches of the current scored candidate object.It is assumed that no pallets are closer than a predetermined distance(e.g., 15 inches) to one another. Thus, if there are more than onescored candidate object closer than that predetermined distance to aparticular current cutout value, the image analysis computer 110identifies, in step 2924, the cutout candidate (i.e., those n scoredcandidate objects determined in step 2912) with a height Y_(W) closestto the current cutout value and uses the attributes from theExistingObjectlist record for this identified scored candidate object topotentially update the current cutout value, cutout[j].

In particular, in step 2926, the Y_(W) value for the identified closestcutout candidate is used to set a value for best_Cutout[j]. This valuefor best_Cutout[j] represents a potentially more accurate value forcutout[j] than its current value. Because step 2926 may potentially berepeated for each of the n different cutout candidates, the value forbest_Cutout[j] may change during later iterations of the loop initiatedin step 2916. Because cutout[j] also has an associated Tag_(cutout) [j],there is also a value best_cut_tag[j] that is set to correspond to theTag_(Object) value of the closest identified cutout candidate that wasused to set best_Cutout[j].

Thus, in step 2924, the image analysis computer 110 identifies thecutout candidate with its Y_(W) value being closest to the currentcutout, cutout[j] identified in step 2922. If, for example, the Y_(W)value for the current cutout candidate is within 15 inches of cutout[2]as determined in step 2922, then the image analysis computer 110determines which of the cutout candidates (from step 2912) has arespective Y_(W) value that is closest to the current value ofcutout[2]. Once this closest cutout candidate is identified, then itsY_(W) value is used to set best_Cutout[2] which is a potentially bettervalue for cutout[2]. For example, best_Cutout[2] can rely partly on thecurrent value for cutout[2] and partly on the identified Y_(W) value.One possible formula for setting the value for best_Cutout[2], as shownin step 2926, may be:best_Cutout[2]=0.5x(cutout[2]+Y _(W))

In step 2928 the image analysis computer 110 determines if the cutoutidentified in step 2922 is the first cutout. As described above, thethree cutouts may be stored as an array (cutout [0], cutout [1] andcutout [2]) and their respective tag values Tag_(cutout) may also bestored as an array (Tag_(cutout)[0], Tag_(cutout)[1] andTag_(cutout)[2]). The image analysis computer 110 can determine if theone cutout[j] identified in step 2922 is the first cutout by checkingits index value in the stored array. If the identified cutout is not thefirst cutout, then the image analysis computer 110 continues with step2932, where cutoutfound[j] is set equal to 1, and then continues theloop started in step 2918. If the identified cutout is the first cutout,then in step 2930 a prior confidence value previously calculated andassociated with this first cutout is adjusted based on the compositeobject score, Score_(Object), of the cutout candidate identified in step2924 as being closest to the one cutout value. For example, the“confidence” value can be calculated according to:confidence=0.5(prior confidence+composite object score (i.e.,Score_(Object)))

Once the confidence value is calculated, the image analysis computer 110continues with step 2932 to continue the loop started in step 2918. Instep 2932, the image analysis computer 110 sets a value forcutoutfound[j] equal to 1, as noted above, so as to indicate that step2922 was satisfied for cutout[j]. The image analysis computer 110continues with step 2950 to move to the next cutout value (i.e.,cutout[j+1]) if available.

If, in step 2920, the current cutout[j] is identified as not being set,then the image analysis computer 110, in step 2938, determines if theY_(W) of the current object, i.e., the scored candidate object of thecurrent iteration of the loop initiated in step 2916, is greater thanthe fork height. If not, then the current loop for the current cutoutvalue, cutout[j], is complete and the index value “j” is incremented instep 2950 so that a new iteration with a new cutout value can begin instep 2918 if the incremented value of j is less than three, for example,in step 2952. When comparing the Y_(w) value with the fork height instep 2938, it may be beneficial to add a relatively small predeterminedvalue to the fork height so that, for example the image analysiscomputer 110 might determine whether or not:

Y_(w)>(fork height+5 inches) rather than making a strict comparison withthe exact fork height value.

If, however, the criteria of step 2938 are satisfied, then the imageanalysis computer 110 continues with step 2940. In step 2940, the imageanalysis computer 110 determines if the first cutout is between zero andthe Y_(W) value of the particular scored candidate object, or cutoutcandidate, being utilized in this particular iteration of step 2916. Ifso, then the image analysis computer 110, in step 2942, sets the secondcutout value to equal Y_(W) for this particular scored candidate objectand sets the second cutout tag value, Tag_(cutout)[1], to theTag_(Object) value for this particular cutout candidate. The imageanalysis computer then moves to step 2950 so that a new iteration with anew cutout value can begin in step 2918 if the incremented value of j isless than three, for example, in step 2952. If the criteria of step 2940are not satisfied, then the image analysis computer 110 proceeds to step2944.

In step 2944, the image analysis computer 110 determines if the firstcutout is greater than Y_(W) of the particular scored candidate object,or cutout candidate, being utilized in this particular iteration of step2916. If so, then in step 2946, the image analysis computer 110 shiftsthe three cutouts right such that the third cutout value is set to thecurrent second cutout value, the second cutout value is set to thecurrent first cutout value, and the first cutout value is set equal toY_(W). The image analysis computer also shifts the corresponding first,second and third Tag_(cutout) values in a similar manner. In addition,the image analysis computer 110 also sets a confidence score for the newfirst cutout value based on a composite object score for the currentlyconsidered scored candidate object, or cutout candidate, according to:confidence=Score_(Object)

Once the cutout values and their respective associated tag values areshifted in step 2946 and the confidence score set, the image analysiscomputer 110 continues with step 2950 so that a new iteration with a newcutout value can begin in step 2918 if the incremented value of j isless than three, for example, in step 2952. If, in step 2944, the imageanalysis computer 110 determines that the first cutout value is notgreater than Y_(W), then the analysis computer 110, in step 2948, setsthe first cutout to equal the Y_(W) value of the particular cutoutcandidate being utilized in this particular iteration of step 2916, setsthe first cutout tag value, Tag_(cutout)[0], equal to the Tag_(Object)value for this particular cutout candidate, sets the “confidence” valuefor the first cutout to equal the composite object score,Score_(Object), for this particular cutout candidate, and continues withstep 2950 so that a new iteration with a new cutout value can begin instep 2918 if the incremented value of j is less than three, for example,in step 2952. Step 2948 is typically reached when the first cutout valueequals “0”. This can occur when all the cutout values are initially setto equal “0” or if a right shift of cutout values happens, in step 2908,results in a value of “zero” being assigned to the first cutout value.Eventually, repeated execution of step 2950 will result in the indexvalue, j, for the loop initiated in step 2918 equaling the number ofcutout values (e.g., three) in step 2952. In this instance, the loop ofstep 2918 is completed and, in step 2954, a new cutout candidate indexvalue i is identified, in step 2956, for a next iteration of the loopinitiated in step 2916. Eventually, i is incremented in step 2956 untilit exceeds the number n of cutout candidates in step 2956. In thisinstance, the loop of step 2916 is completed and the image analysiscomputer continues with step 2958.

In step 2958, a loop through the different cutout values is initiated(e.g., j=0 to 2). First, in step 2960, the image analysis computer 110determines if cutoutfound[j] is not equal to zero. This criterion istrue when step 2922 was satisfied at least once for cutout[j]. As aresult, a potentially better value for cutout[j] was also calculated asbest_Cutout[j] which has a corresponding tag value of best_cut_tag[j].Accordingly, in step 2962, cutout[j] and Tag_(cutout)[j] are updated toreflect the values of best_Cutout[j] and best_cut_tag[j], respectively.

Once step 2962 is completed, or if cutout[j] equals “0”, in step 2960,the image analysis computer 110 increments j in step 2964 and loops backto step 2958 if there are more cutouts remaining as determined by step2966. Once the loop of step 2958 completes for all the cutouts, theimage analysis computer 110 continues with step 2970.

If no possible scored candidate objects were identified in step 2912 forthe current fork height, then the image analysis computer 110 continueswith step 2968. In step 2968, the confidence value that is associatedwith the first cutout value is reduced. For example, a new confidencevalue can be calculated according to:confidence=0.5(confidence).

Once the confidence score is adjusted in step 2968 or once all the stepsof the loop of 2958 completes, the image analysis computer 110 continueswith step 2970. In particular, if the image analysis computer 110determines, in step 2970, that the confidence value for the first cutoutis above a predetermined threshold (e.g., 0.5), then a counterassociated with the confidence value may, for example, be decremented instep 2972. For example, if the counter is set equal to three and isdecremented each time the confidence value is above the predeterminedthreshold, then the value of the counter will be lower the more times ahigher confidence value for the first cutout is encountered. One ofordinary skill will recognize that this counter value may vary and may,for example, be greater than three (e.g., five) or less than three(e.g., one).

If the confidence value is not above the predetermined threshold, asdetermined in step 2970, or once the counter is adjusted, as in step2972, then the image analysis computer 110 determines if the firstcutout value is greater than zero, in step 2974. If so, then an adjustedfirst cutout value for the first cutout is calculated by the imageanalysis computer 110 in step 2976. For example, the adjusted firstcutout value can be calculated according to:adjusted value=cutout[0]+1.5 inches

wherein a predetermined offset value (e.g., 1.5 inches) is added to thevalue for cutout[0]. Once the adjusted value is calculated in step 2976or if the first cutout value is not greater than zero, as determined instep 2974, then the image analysis computer 110 continues with step2978. In step 2978, the image analysis computer 110 determines if thecounter is more than a predetermined value. If, for example, the countervalue is greater than zero, then a minimum number of “confidence” valuesover the predetermined threshold have not occurred (see step 2970 and2972). Under these circumstances, the image analysis computer 110, instep 2980, resets the adjusted value to equal zero and repeats theprocess by returning to step 2904.

If however, the counter value indicates that a minimum number of“confidence” values have occurred that are over the predeterminedthreshold, then the image analysis computer 110, in step 2982,communicates the “adjusted” value (e.g., first cutout value+1.5 inches)to the vehicle computer 50 as a new first cutout location value or stopheight and may also communicate the second and third cutout values aswell. The process then repeats by returning to step 2904.

The above description assumed that the forks are being raised upwardsand that the operator desires for the forks to stop at the next highercutout. However, one of ordinary skill will recognize that the algorithmof FIGS. 29A-29F may be modified to also operate appropriately when theforks are traveling downward and the operator desires for the forks tostop at the next lower cutout.

The accuracy with which scored candidate objects may be detected andtracked through a series of image frames can be improved by performingintrinsic and extrinsic calibration of the camera. As mentionedpreviously, an intrinsic camera matrix has the form:

$\lbrack K\rbrack = \begin{bmatrix}f_{x} & \gamma & x_{0} \\0 & f_{y} & y_{0} \\0 & 0 & 1\end{bmatrix}$

-   -   where:    -   (x₀, y₀) is the camera center 2714 and is the pixel location        coordinates on the image plane 2712 where the camera center (or        the camera coordinate system origin) 2708 projects according to        the pinhole camera model;    -   f_(x) is the focal length, f, expressed in x-direction, or        horizontal, pixel-related units; and    -   f_(y) is the focal length, f, expressed in y-direction, or        vertical, pixel related units, and    -   γ represents a skew factor between the x and y axes, and is        often “0”.

The values for the elements of this matrix can be determined using knowncamera resectioning techniques and can be performed as part of themanufacturing process of the camera. As for extrinsic calibration of theimaging camera 130, other techniques are useful for determining valuesfor the previously discussed extrinsic camera matrix [R_(cam)|T_(cam)].As these values may depend on the specific manner in which the camera ismounted relative to the vehicle 20, extrinsic calibration maybeneficially be performed at a site where the vehicle 20 will be used.For example, values may be determined for:

-   -   t_(ybias) (the fixed vertical offset between the imaging camera        130 and the location of the forks 42A and 42B),    -   t_(xbias) (the fixed horizontal offset between the imaging        camera 130 and the location of the world coordinate origin        2706),    -   θ_(x) (the rotation of the camera coordinate system relative to        the X-axis 2707, referred to as an elevation angle, designated        θ_(e) in FIG. 27A). Ideally the elevation angle may be        calibrated when the forks 42A and 42B have a tilt angle of zero.        As noted above, if the tilt angle θ_(ef) of the forks 42A and        42B, which can be provided by a tilt angle sensor in        communication with the vehicle computer 50, is not zero, then        the overall elevation angle θ_(e)=(θ_(ec)+θ_(ef)) of the imaging        camera 130 will include a portion θ_(ef) due to the tilting of        the forks 42A and 42B and a portion θ_(ec) due to the tilt of        the camera 130,    -   θ_(y) (the rotation of the camera coordinate system relative to        the Y-axis 2711, referred to as a pitch angle), and    -   θ_(z) (the rotation of the camera coordinate system relative to        the Z-axis 2705, referred to as a deflection angle).

An initial estimate can be assigned to these values in order to beginthe extrinsic camera calibration process. For example, when consideringonly an elevation angle, an estimated initial value of θ_(x) may be 0degrees and an initial estimated value for t_(ybias) may be a physicallymeasured height of the imaging camera 130 from the forks 42A and 42B.The horizontal offset t_(xbias), the pitch angle and deflection anglemay be considered to be equal to zero. Using these initial estimatedvalues more than one scored candidate object (i.e., a potential pallet)may be acquired in an image of a known, or measured, world geometry. Forexample, based on a respective, manually pre-measured distance andheight of more than one pallet relative to the world coordinate origin2706, which, as noted above, is located at a known location on thevehicle 10, corresponding world coordinate locations (X_(w), Y_(w),Z_(w)) can be determined (where the value of X_(w) may not necessarilybe measured when only an elevation angle is non-zero and camera has notranslation offset, as described more fully below, in the X-direction(i.e., t_(xbias)=0). Alternatively, a similar technique can beaccomplished which uses respective measurement information about asingle potential pallet from multiple image frames taken from differentcamera positions.

FIG. 30 depicts an image frame in a series of image frames for oneexample scenario in which three potential pallet locations 3002, 3004,3006 have been captured by the imaging camera 130 and projected onto animage plane 3010 as three respective pixel locations 3012, 3014, and3016. It is also possible to acquire calibration data from multipleviews of chessboards where the corner locations substitute for the lowerstringer point. As discussed earlier, these pixel locations may becorrected for distortion such that the locations 3012, 3014, and 3016are in undistorted image space. In the depicted scenario, the followingvalues have been determined:

the translation of the forks, t_(y), when the image frame 3010 iscaptured can be determined from the vehicle computer 50;

the first potential pallet location 3002 is projected to undistortedimage pixel location (x₁, y₁) 3012 and is measured relative to a worldcoordinate origin to have a corresponding world coordinate location of(X₁, Y₁, Z₁);

the second potential pallet location 3004 is projected to undistortedimage pixel location (x₂, y₂) 3014 and is measured relative to a worldcoordinate origin to have a corresponding world coordinate location of(X₂, Y₂, Z₂); and

the third potential pallet location 3006 is projected to undistortedimage pixel location (x₃, y₃) 3016 and is measured relative to a worldcoordinate origin to have a corresponding world coordinate location of(X₃, Y₃, Z₃).

In a configuration where there is only an elevation angle (i.e., θ_(x)),the second linear equation of Equation 23 provides Equation 41A (whereθ_(x) denotes the elevation angle):y _(row)(Y _(w) sin θ_(x) +Z _(w) cos θ_(x))−Y _(w)(f _(y) cos θ_(x) +y₀ sin θ_(x))−Z _(w)(−f _(y) sin θ_(x) +y ₀ cos θ_(x))=f _(y)(t _(y) +t_(ybias))This equation can be rearranged to provide a first equation useful inperforming the extrinsic camera calibration. In particular, Equation 41Acan be rearranged into Equation 41B:cos θ_(x)(−f _(y) Y _(w)+(y _(row) −y ₀)Z _(w))+sin θ_(x)((y _(row) −y₀)Y _(w) +f _(y) Z _(w))=f _(y)(t _(y) +t _(ybias))As mentioned previously, the elevation angle θ_(x), can include aportion θ_(ec) that is from a tilt angle of the imaging camera 130 and aportion θ_(ef) that is from angle of the forks 42A and 42B such that:θx=θ _(ec)+θ_(eƒ)A second equation can be derived from the well-known trigonometricalprinciple of Equation 42:cos²θ_(ec)+sin²θ_(ec)=1.Using these two equations, appropriate values for t_(ybias) and θ_(x)can be determined using an iterative process as described below. Ingeneral, the iterative process first defines two residual functionsbased on Equation 41B and Equation 42. In particular, the two residualfunctions are defined by Equation 43A:f=f _(y)(t _(y) +t _(ybias))−cos θ_(x)(−f _(y) Y _(w)+(y _(row) −y ₀)Z_(w))−sin θ_(x)((y _(row) −y ₀)Y _(w) +f _(y) Z _(w))However, Equation 43A can relate to more than one potential pallet (orscored candidate object); thus, it can be more generally written asEquation 43B:f(i)=f _(y)(t _(y) +t _(ybias))−cos θ_(x)(−f _(y) Y _(i)+(y _(i) −y ₀)Z_(i))−sin θ_(x)((y _(i) −y ₀)Y _(i) +f _(y) Z _(i))where i is an index used to distinguish between different scoredcandidate objects (e.g., 3002, 3004, 3006) that may be observed in theimage frame 3010. Each such scored candidate object, in Equation 43B,has a respective residual equation f(i) and has a respective manuallymeasured world coordinate location (X_(i), Y_(i), Z_(i)) and an imagepixel location height value of y_(i). The format of Equation 43B can besimplified by writing it as Equation 43C:f(i)=f _(y)(t _(y) +t _(ybias))−cos θ_(x)(q _(i))−sin θ_(x)(r _(i))where:q _(i)=+(−f _(y) Y _(i)+(y _(i) −y ₀)Z _(i))r _(i)=((y _(i) −y ₀)Y _(i) +f _(y) Z _(i))Using the two geometrical identities:cos(θ_(x))=cos(θ_(ec)+θ_(ef))=cos(θ_(ec))cos(θ_(ef))−sin(θ_(ec))sin(θ_(ef))sin(θ_(x))=sin(θ_(ec)+θ_(ef))=sin(θ_(ec))cos(θ_(ef))+cos(θ_(ec))sin(θ_(ef))allows Equation 43C to be written as Equation 43D:f(i)=f _(y)(t _(y) +t_(ybias))−[cos(θ_(ec))cos(θ_(ef))−sin(θ_(ec))sin(θ_(ef))](q_(i))−[sin(θ_(ec))cos(θ_(ef))+cos(θ_(ec))sin(θ_(ef))](r _(i))

The second residual equation based on Equation 42 is defined by Equation44:g _(e)=1−cos²θ_(ec)−sin²θ_(ec)

By finding the respective roots of each residual equation (i.e., thosevalues of the variables that result in f(i) and g_(e) equaling zero),appropriate values for determining the extrinsic camera calibrationvalues can be calculated. In the above notation, f(i) represents theresidual equation for the scored candidate object i and does not denotethat f is a function of an independent variable i.

In general, a function w can be evaluated at a point (n₀+e) using aTaylor, or power, series where:w(n ₀ +e)=w(n ₀)+w′(n ₀)e+½w″(n ₀)e ² . . . .

Taking just the first two terms of the series provides an approximationof w where:w(n ₀ +e)≈w(n ₀)+w′(n ₀)eIn this Taylor approximation, n₀ may be considered as an initial guessat a root of the function w and e is an initial offset intended tolocate a value closer to an actual root of w than n₀. Setting w(n₀+e)=0and solving for “e”, provides an initial estimated value for the offset“e”. This use of the Taylor series can be iteratively performed suchthat each previous calculation of “e” is used in the next iteration ofsolving for the root of w according to the Newton-Rhapson method.

In other words, using a current guess, n(t), for the root of w provides:0≈w(n(t))+w′(n(t))*e(t)which can then be solved to estimate a value for an offset, or “step”used in the iterative algorithm according to:

${- \frac{w\left( {n(t)} \right)}{w^{\prime}\left( {n(t)} \right)}} \approx {e(t)}$Using the newly calculated estimate for the step e(t), the updatedguess, n(t+1), for the root of w is provided by:n(t+1)=n(t)+e(t)

It is this value n(t+1) that is used in the next iteration ofcalculating the estimated offset, or step, value. This iterative processcan be repeated until e(t) reaches a predetermined threshold value. Inthis way, the Taylor series approximation may be iteratively used tosolve for a root of the function w.

For multivariable functions, such as for example, w(n, m), the first twoterms of a Taylor series approximation around the point (n₀+e₁, m₀+e₂)is formed by taking partial derivatives according to:

${w\left( {{n_{0} + e_{1}},{m_{0} + e_{2}}} \right)} \approx {{w\left( {n_{0},m_{0}} \right)} + \left( {{\frac{\partial{w\left( {n_{0},m_{0}} \right)}}{\partial n}\left( e_{1} \right)} + {\frac{\partial{w\left( {n_{0},m_{0}} \right)}}{\partial m}\left( e_{2} \right)}} \right)}$

For each of the residual equations related to the extrinsic cameracalibration in this first configuration, the variables of interest arecos θ_(ec), sin θ_(ec), and t_(ybias) and each has a respectiveestimated offset p₀, p₁, and p₂. If there are at least two scoredcandidate objects, then using the above-described principles aboutTaylor series approximations provides the following residual equationsfor performing an extrinsic camera calibration:g _(e)=1−cos²θ_(ec)−sin²θ_(ec)f(0)=f_(y)(t _(y) +t_(ybias))−[cos(θ_(ec))cos(θ_(ef))−sin(θ_(ec))sin(θ_(ef))](q₀)−[sin(θ_(ec))cos(θ_(ef))+cos(θ_(ec))sin(θ_(ef))](r ₀)f(1)=f_(y)(t _(y) +t_(ybias))−[cos(θ_(ec))cos(θ_(ef))−sin(θ_(ec))sin(θ_(ef))](q₁)−[sin(θ_(ec))cos(θ_(ef))+cos(θ_(ec))sin(θ_(ef))](r ₁)If there are additional scored candidate objects available, thenadditional residual equations may be used as well; these additionalresidual equations would have the form of:f(i)=f _(y)(t _(y) +t_(ybias))−[cos(θ_(ec))cos(θ_(ef))−sin(θ_(ec))sin(θ_(ef))](q_(i))−[sin(θ_(ec))cos(θ_(ef))+cos(θ_(ec))sin(θ_(ef))](r _(i))where i refers to an index value that uniquely identifies the scoredcandidate object.

The respective Taylor series approximations of the residual equationsare Equation 46A:g _(e)≈1−cos²θ_(ec)−sin²θ_(ec)−2 cos θ_(ec) *p ₀−2 sin θ_(ec) *p ₁f(0)≈f _(y)(t _(y) +t_(ybias))−[cos(θ_(ec))cos(θ_(ef))−sin(θ_(ec))sin(θ_(ef))](q₀)−[sin(θ_(ec))cos(θ_(ef))+cos(θ_(ec))sin(θ_(ef))](r ₀)−[q ₀cos(θ_(ef))+r ₀ sin(θ_(ef))]*p ₀ −[−q ₀ sin(θ_(ef))+r ₀ cos(θ_(ef))]*p ₁+f _(y) *p ₂f(1)≈f _(y)(t _(y) +t_(ybias))−[cos(θ_(ec))cos(θ_(ef))−sin(θ_(ec))sin(θ_(ef))](q₁)−[sin(θ_(ec))cos(θ_(ef))+cos(θ_(ec))sin(θ_(ef))](r ₁)−[q ₁cos(θ_(ef))+r ₁ sin(θ_(ef))]*p ₀ −[−q ₁ sin(θ_(ef))+r ₁ cos(θ_(ef))]*p ₁+f _(y) *p ₂

Setting each of the Equations from Equation 46A equal to zero to solvefor the respective roots provides Equation 47A:2 cos θ_(ec) *p ₀+2 sin θ_(ec) *p ₁≈1−cos²θ_(ec)−sin²θ_(ec)[q ₀ cos(θ_(ef))+r ₀ sin(θ_(ef))]*p ₀ +[−q ₀ sin(θ_(ef))+r ₀cos(θ_(ef))]*p ₁ −f _(y) *p ₂ ≈f _(y)(t _(y) +t_(ybias))−[cos(θ_(ec))cos(θ_(ef))−sin(θ_(ec))sin(θ_(ef))](q₀)−[sin(θ_(ec))cos(θ_(ef))+cos(θ_(ec))sin(θ_(ef))](r ₀)[q ₁ cos(θ_(ef))+r ₁ sin(θ_(ef))]*p ₀ +[−q ₁ sin(θ_(ef))+r ₁cos(θ_(ef))]*p ₁ −f _(y) *p ₂ ≈f _(y)(t _(y) +t_(ybias))−[cos(θ_(ec))cos(θ_(ef))−sin(θ_(ec))sin(θ_(ef))](q₁)−[sin(θ_(ec))cos(θ_(ef))+cos(θ_(ec))sin(θ_(ef))](r ₁)The three equations of Equation 47A can be written in matrix notation asEquation 48A:

$\mspace{20mu}{{\begin{bmatrix}{2\;\cos\;\theta_{ec}} & {2\;\sin\;\theta_{ec}} & 0 \\{{q_{0}\cos\;\theta_{ef}} + {r_{0}\sin\;\theta_{ef}}} & {{{- q_{0}}\sin\;\theta_{ef}} + {r_{0}\cos\;\theta_{ef}}} & {- f_{y}} \\{{q_{1}\cos\;\theta_{ef}} + {r_{1}\sin\;\theta_{ef}}} & {{{- q_{1}}\sin\;\theta_{ef}} + {r_{1}\cos\;\theta_{ef}}} & {- f_{y}} \\\vdots & \vdots & \vdots\end{bmatrix}\begin{bmatrix}p_{0} \\p_{1} \\p_{2}\end{bmatrix}} = \begin{bmatrix}g_{e} \\{f(0)} \\{f(1)} \\\vdots\end{bmatrix}}$   where:   q_(i) = (−f_(y)Y_(i) + (y_(i) − y₀)Z_(i))  r_(i) = ((y_(i) − y₀)Y_(i) + f_(y)Z_(i))f(i) = f_(y)(t_(y) + t_(ybias)) − [cos (θ_(ec))cos (θ_(ef)) − sin (θ_(ec))sin (θ_(ef))](q_(i)) − [sin (θ_(ec))cos (θ_(ef)) + cos (θ_(ec))sin (θ_(ef))](r_(i))  g_(e) = 1 − cos²θ_(ec) − sin²θ_(ec)

As mentioned, there may be more than just two scored candidate objectswithin the image frame 3010 and, as a result, Equation 48A would haveadditional rows of values q_(i), r_(i), and f_(i). The general form ofEquation 48A is that shown in Equation 49A:

$\lbrack J\rbrack = {\begin{bmatrix}p_{0} \\p_{1} \\p_{2}\end{bmatrix} = \lbrack F\rbrack}$${{where}{\text{:}\lbrack J\rbrack}} = {{\begin{bmatrix}{2\;\cos\;\theta_{ec}} & {2\;\sin\;\theta_{ec}} & 0 \\{{q_{0}\cos\;\theta_{ef}} + {r_{0}\sin\;\theta_{ef}}} & {{{- q_{0}}\sin\;\theta_{ef}} + {r_{0}\cos\;\theta_{ef}}} & {- f_{y}} \\{{q_{1}\cos\;\theta_{ef}} + {r_{1}\sin\;\theta_{ef}}} & {{{- q_{1}}\sin\;\theta_{ef}} + {r_{1}\cos\;\theta_{ef}}} & {- f_{y}} \\\vdots & \vdots & \vdots\end{bmatrix}\mspace{14mu}{{and}\lbrack F\rbrack}} = \begin{bmatrix}g_{e} \\{f(0)} \\{f(1)} \\\vdots\end{bmatrix}}$

which can be manipulated to form Equation 49B:

$\begin{bmatrix}p_{0} \\p_{1} \\p_{2}\end{bmatrix} = {{\left( {\lbrack J\rbrack^{T}\lbrack J\rbrack} \right)^{- 1}\lbrack J\rbrack}^{T}\lbrack F\rbrack}$Accordingly, Equation 49B reveals that respective observations ofmultiple pallets and a respective estimate for cos θ_(ec), sin θ_(ec),and t_(ybias) can be used to provide a least squares solution for thevector of values

$\begin{bmatrix}p_{0} \\p_{1} \\p_{2}\end{bmatrix}.$This vector of values

$\quad\begin{bmatrix}p_{0} \\p_{1} \\p_{2}\end{bmatrix}$can be used to iteratively update the respective estimate for cosθ_(ec), sin θ_(ec), and t_(ybias) according to Equation 50:cos θ_(ec)(t+1)=cos θ_(ec)(t)+p ₀sin θ_(ec)(t+1)=sin θ_(ec)(t)+p ₁t _(ybias)(t+1)=t _(ybias)(t)+p ₂The updated estimates for cos θ_(ec), sin θ_(ec), and t_(ybias) can thenbe used in Equation 49B to find new values for

$\begin{bmatrix}p_{0} \\p_{1} \\p_{2}\end{bmatrix}.$This iterative process may be repeated until each of the calculatedvalues for

$\quad\begin{bmatrix}p_{0} \\p_{1} \\p_{2}\end{bmatrix}$are smaller than a predetermined threshold. For example, thispredetermined threshold may be 10⁻⁶. Once the iterative processcompletes, a respective value for cos θ_(ec), sin θ_(ec), and t_(ybias)has been calculated according to Equation 50 and the camera'scontribution to the elevation angle, θ_(ec), can be determined from

$\theta_{ec} = {{\tan^{- 1}\left( \frac{\sin\;\theta_{ec}}{\cos\;\theta_{ec}} \right)}.}$The camera's contribution to the elevation angle, θ_(ec), and t_(ybias)comprise the extrinsic camera calibration values for this embodiment.

In the configuration described above, using only one of the linearequations from Equation 23 was sufficient (along with g_(e)) in orderfor multiple pallet observations to be used to solve for the twoextrinsic camera calibration values θ_(ec) and t_(ybias). However, in aconfiguration in which only an elevation angle θ_(x) is present, theremay still be instances in which t_(ybias)≠0. In this instance, manuallymeasuring the horizontal offset between a location of the imaging camera130 and a world coordinate origin 2706 location may provide sufficientaccuracy for this value. However, performing extrinisc cameracalibration to determine a calibration value for t_(xbias) can also beaccomplished as well but will utilize additional equations than those ofEquation 48A. A full explanation of how t_(xbias) may be defined tosimplify physical measurement of an X_(w) coordinate for a particularpallet location is provided below with respect to a camera configurationin which there is both an elevation angle θ_(x) and a deflection angleθ_(z).

In a configuration in which there is no deflection angle θ_(z) but thevalue for t_(xbias)≠0 the first linear equation of Equation 23 was givenby:−X _(w) f _(x) −Y _(w) x ₀ sin θ_(x) −Z _(w) x ₀ cos θ_(x) +x _(i)(Y_(w) sin θ_(x) +Z _(w) cos θ_(x))−f _(x)(t _(xbias))=0

which can be rearranged to provide an additional residual equation for aparticular pallet location i that is useful for determining t_(xbias):f*(i)=X _(w) f _(x) +f _(x)(t _(xbias))−((x _(i) −x ₀)Z _(w))cosθ_(x)−((x _(i) −x ₀)Y _(w))sin θ_(x)where the notation f*(i) is used for convenience to distinguish thisadditional residual equation from the previously discussed residualequation f(i). This additional residual equation can be written asEquation 43E:f*(i)=X _(w) f _(x) +f _(x)(t _(xbias))−cos θ_(x)(q _(i)*)−sin θ_(x)(r_(i)*)where:q _(i)*=((x _(i) −x ₀)Z _(w))r _(i)*=((x _(i) −x ₀)Y _(w))Again, using the two geometrical identities:cos(θ_(x))=cos(θ_(ec)+θ_(ef))=cos(θ_(ec))cos(θ_(ef))−sin(θ_(ec))sin(θ_(ef))sin(θ_(x))=sin(θ_(ec)+θ_(ef))=sin(θ_(ec))cos(θ_(ef))+cos(θ_(ec))sin(θ_(ef))allows Equation 43E to be written as Equation 43F:f*(i)=X _(w) f _(x) +f _(x)(t_(xbias))−[cos(θ_(ec))cos(θ_(ef))−sin(θ_(ec))sin(θ_(ef))](q_(i)*)−[sin(θ_(ec))cos(θ_(ef))+cos(θ_(ec))sin(θ_(ef))](r _(i)*)

Using this additional residual equation related to the extrinsic cameracalibration in this modified first configuration, the variables ofinterest are cos θ_(ec), sin θ_(ec), t_(ybias) and t_(xbias) and eachhas a respective estimated offset p₀, p₁, p₂, and p₃. If there are atleast two scored candidate objects, then using the above-describedprinciples about Taylor series approximations provides the followingresidual equations for performing an extrinsic camera calibration:g _(e)=1−cos²θ_(ec)−sin²θ_(ec)f(0)=f _(y)(t _(y) +t _(ybias))−[cos(θ_(ec))cos(θ_(ef))−sin(θ_(ec)sin(θ_(ef))](q ₀)−[sin(θ_(ec))cos(θ_(ef))+cos(θ_(ec))sin(θ_(ef))](r ₀)f*(0)=X _(w) f _(x) +f _(x)(t_(xbias))−[cos(θ_(ec))cos(θ_(ef))−sin(θ_(ec))sin(θ_(ef))](q₀*)−[sin(θ_(ec))cos(θ_(ef))+cos(θ_(ec))sin(θ_(ef))](r ₀*)f(1)=f _(y)(t _(y) +t_(ybias))−[(cos(θ_(ec))cos(θ_(ef))−sin(θ_(ec))sin(θ_(ef))](q₁)−[sin(θ_(ec))cos(θ_(ef))+cos(θ_(ec))sin(θ_(ef))](r ₁)f*(1)=X _(w) f _(x) +f _(x)(t_(xbias))−[cos(θ_(ec))cos(θ_(ef))−sin(θ_(ec))sin(θ_(ef))](q₁*)−[sin(θ_(ec))cos(θ_(ef))+cos(θ_(ec))sin(θ_(ef))](r ₁*)  Equation 45B:If there are additional scored candidate objects available, thenadditional residual equations may be used as well; these additionalresidual equations would have the form of:f*(i)=X _(w) f _(x) +f _(x)(t_(xbias))−[cos(θ_(ec))cos(θ_(ef))−sin(θ_(ec))sin(θ_(ef))](q_(i)*)−[sin(θ_(ec))cos(θ_(ef))+cos(θ_(ec))sin(θ_(ef))](r _(i)*)where i refers to an index value that uniquely identifies the scoredcandidate object.

The respective Taylor series approximations of the residual equations ofEquation 45B are Equation 46B:g _(e)≈1−cos²θ_(ec)−sin²θ_(ec)−2 cos θ_(ec) *p ₀−2 sin θ_(ec) *p ₁f(0)≈f _(y)(t _(y) +t_(ybias))−[cos(θ_(ec))cos(θ_(ef))−sin(θ_(ec))sin(θ_(ef))](q₀)−[sin(θ_(ec))cos(θ_(ef))+cos(θ_(ec))sin(θ_(ef))](r ₀)−[q ₀cos(θ_(ef))+r ₀ sin(θ_(ef))]*p ₀ −[−q ₀ sin(θ_(ef))+r ₀ cos(θ_(ef))]*p ₁+f _(y) *p ₂f*(0)≈X _(w) f _(x) +f _(x)(t_(xbias))−[cos(θ_(ec))cos(θ_(ef))−sin(θ_(ec))sin(θ_(ef))](q₀*)−[sin(θ_(ec))cos(θ_(ef))+cos(θ_(ec))sin(θ_(ef))](r ₀*)−[q₀*cos(θ_(ef))+r ₀*sin(θ_(ef))]*p ₀ −[−q ₀*sin(θ_(ef))+r ₀*cos(θ_(ef))]*p₁ +f _(x) *p ₃f(1)≈f _(y)(t _(y) +t_(ybias))−[cos(θ_(ec))cos(θ_(ef))−sin(θ_(ec))sin(θ_(ef))](q₁)−[sin(θ_(ec))cos(θ_(ef))+cos(θ_(ec))sin(θ_(ef))](r ₁)−[q ₁cos(θ_(ef))+r ₁ sin(θ_(ef))]*p ₀ −[−q ₁ sin(θ_(ef))+r ₁ cos(θ_(ef))]*p ₁+f _(y) *p ₂f*(1)≈X _(w) f _(x) +f _(x)(t_(xbias))−[cos(θ_(ec))cos(θ_(ef))−sin(θ_(ec))sin(θ_(ef))](q₁*)−[sin(θ_(ec))cos(θ_(ef))+cos(θ_(ec))sin(θ_(ef))](r ₁*)−[q₁*cos(θ_(ef))+r ₁*sin(θ_(ef))]*p ₀ −[−q ₁*sin(θ_(ef))+r ₁*cos(θ_(ef))]*p₁ +f _(x) *p ₃

Similar to before, setting each of the Equations from Equation 46B equalto zero to solve for the respective roots provides Equation 47B:2 cos θ_(ec) *p ₀+2 sin θ_(ec) *p ₁≈1−cos²θ_(ec)−sin²θ_(ec)[q ₀ cos(θ_(ef))+r ₀ sin(θ_(ef))]*p ₀ +[−q ₀ sin(θ_(ef))+r ₀cos(θ_(ef))]*p ₁ −f _(y) *p ₂ ≈f _(y)(t _(y) +t_(ybias))−[cos(θ_(ec))cos(θ_(ef))−sin(θ_(ec))sin(θ_(ef))](q₀)−[sin(θ_(ec))cos(θ_(ef))+cos(θ_(ec) sin(θ_(ef))](r ₀)[q ₀*cos(θ_(ef))+r ₀*sin(θ_(ef))]*p ₀ +[−q ₀*sin(θ_(ef))+r₀*cos(θ_(ef))]*p ₁ −f _(y) *p ₃ ≈X _(w) f _(x) +f _(x)(t_(xbias))−[cos(θ_(ec))cos(θ_(ef))−sin(θ_(ec))sin(θ_(ef))](q₀*)−[sin(θ_(ec))cos(θ_(ef))+cos(θ_(ec) sin(θ_(ef))](r ₀*)[q ₁ cos(θ_(ef))+r ₁ sin(θ_(ef))]*p ₀ +[−q ₁ sin(θ_(ef))+r ₁cos(θ_(ef))]*p ₁ −f _(y) *p ₂ ≈f _(y)(t _(y) +t_(ybias))−[cos(θ_(ec))cos(θ_(ef))−sin(θ_(ec))sin(θ_(ef))](q₁)−[sin(θ_(ec))cos(θ_(ef))+cos(θ_(ec) sin(θ_(ef))](r ₁)[q ₁*cos(θ_(ef))+r ₁*sin(θ_(ef))]*p ₀ +[−q ₁*sin(θ_(ef))+r₁*cos(θ_(ef))]*p ₁ −f _(y) *p ₃ ≈X _(w) f _(x) +f _(x)(t_(xbias))−[cos(θ_(ec))cos(θ_(ef))−sin(θ_(ec))sin(θ_(ef))](q₁*)−[sin(θ_(ec))cos(θ_(ef))+cos(θ_(ec) sin(θ_(ef))](r ₁*)The five equations of Equation 47B can be written in matrix notation asEquation 48B:

${\begin{bmatrix}{2\;\cos\;\theta_{ec}} & {2\;\sin\;\theta_{ec}} & 0 & 0 \\{{q_{0}\cos\;\theta_{ef}} + {r_{0}\sin\;\theta_{ef}}} & {{{- q_{0}}\sin\;\theta_{ef}} + {r_{0}\cos\;\theta_{ef}}} & {- f_{y}} & 0 \\{{q_{0}^{*}\cos\;\theta_{ef}} + {r_{0}^{*}\sin\;\theta_{ef}}} & {{{- q_{0}^{*}}\sin\;\theta_{ef}} + {r_{0}^{*}\cos\;\theta_{ef}}} & 0 & {- f_{x}} \\{{q_{1}\cos\;\theta_{ef}} + {r_{1}\sin\;\theta_{ef}}} & {{{- q_{1}}\sin\;\theta_{ef}} + {r_{1}\cos\;\theta_{ef}}} & {- f_{y}} & 0 \\{{q_{1}^{*}\cos\;\theta_{ef}} + {r_{1}^{*}\sin\;\theta_{ef}}} & {{{- q_{1}^{*}}\sin\;\theta_{ef}} + {r_{1}^{*}\cos\;\theta_{ef}}} & 0 & {- f_{x}} \\\vdots & \vdots & \vdots & \vdots\end{bmatrix}\begin{bmatrix}p_{0} \\p_{1} \\p_{2} \\p_{3}\end{bmatrix}} = \begin{bmatrix}g_{e} \\{f(0)} \\{f^{*}(0)} \\{f(1)} \\{f^{*}(1)} \\\vdots\end{bmatrix}$   where:   q_(i) = (−f_(y)Y_(i) + (y_(i) − y₀)Z_(i))  r_(i) = ((y_(i) − y₀)Y_(i) + f_(y)Z_(i))  q_(i)^(*) = ((x_(i) − x₀)Z_(w))   r_(i)^(*) = ((x_(i) − x₀)Y_(w))f(i) = f_(y)(t_(y) + t_(ybias)) − [cos (θ_(ec))cos (θ_(ef)) − sin (θ_(ec))sin (θ_(ef))](q_(i)) − [sin (θ_(ec))cos (θ_(ef)) + cos (θ_(ec))sin (θ_(ef))](r_(i))f^(*)(i) = X_(w)f_(x) + f_(x)(t_(xbias)) − [cos (θ_(ec))cos (θ_(ef)) − sin (θ_(ec))sin (θ_(ef))](q_(i)^(*)) − [sin (θ_(ec))cos (θ_(ef)) + cos (θ_(ec))sin (θ_(ef))](r_(i)^(*))  and  g_(e) = 1 − cos²θ_(ec) − sin²θ_(ec)

Equation 48B can then be used in a similar manner to Equations 48A, 49A,49B, and 50 to iteratively solve for the three extrinsic cameracalibration values: the camera's contribution to the elevation angle,θ_(ec), t_(xbias), and t_(ybias) which comprise the extrinsic cameracalibration values for this modified first configuration having anon-zero elevation angle θ_(x) and X-direction offset t_(xbias).

When both an elevation angle θ_(x) and a deflection angle θ_(z) arepresent, then all three of the respective world coordinate values(X_(w), Y_(w), Z_(w)) for each pallet location may be used to calculatethe extrinsic camera calibration values. In particular, the X_(w) valuefor a particular pallet location may be considered as being relative toa reference pallet location such as, for example, the bottom most palletlocation (e.g., 3006 of FIG. 30). In order to do so, the horizontaloffset value, t_(xbias), may be defined in a particular manner to aidthe calibration process. As discussed earlier, the imaging camera 130may be attached to the vehicle frame in such a way that it has anX-coordinate value offset from the world coordinate origin 2706 by abias value (i.e., tc_(x)=t_(xbias)). During calibration, the vehicle 20can be carefully positioned so that the location on the vehicle 20 thathas been designated as the world origin 2706 is substantially aligned inthe X-direction with the bottom most pallet location (e.g., 3006 of FIG.30). As a result, a horizontal world coordinate X₃ may be defined asequal to zero and the horizontal world coordinate X₂ may be obtained bymeasuring a horizontal distance between a lower center of a centralpallet stringer at pallet location 3004 and a center of the lowercentral pallet stringer at pallet location 3006. A horizontal coordinateX₁ for pallet location 3002 may be measured in a similar fashion. As aresult, the process described below will permit a calibration value fort_(xbias) to be determined.

When a deflection angle is also present, the linear equations ofEquation 24B can be rearranged to provide the residual equations, for ascored candidate object i, of Equation 51A:f ₁(i)=((x _(i) −x ₀)sin θ_(x) sin θ_(z) −f _(x) cos θ_(z))X _(i)+((x_(i) −x ₀)cos θ_(z) sin θ_(x))Y _(i)+((x _(i) −x ₀)cos θ_(x))Z _(i) −f_(x) t _(xbias) f ₂(i)=(((y _(i) −y ₀)sin θ_(x) −f _(y) cos θ_(x))sinθ_(z))X _(i)+(((y _(i) −y ₀)sin θ_(x) −f _(y) cos θ_(x))cos θ_(z))Y_(i)+((y _(i) −y ₀)cos θ_(x) +f _(y) sin θ_(x))Z _(i) −f _(y)(t _(y) +t_(ybias))The two other residual equations are provided by Equation 51B:g _(e)=cos²θ_(ec)+sin²θ_(ec)−1g _(d)=cos²θ_(z)+sin²θ_(z)−1

where θ_(x)=θ_(ec)+θ_(ef).

The Newton-Rhapson iterative method described above with respect to theelevation angle-only scenario may be applied in a similar manner to thepresent scenario where the extrinsic camera parameters to be calculatedare a deflection angle θ_(z), a camera elevation angle, θ_(ec), anx-bias value, t_(xbias), and a y bias value, t_(ybias). For at least twopotential pallet locations in the image plane 3010, a matrix [F′] can bedefined as Equation 52:

$F^{\prime} = \begin{bmatrix}g_{e} \\g_{d} \\{f_{1}(0)} \\{f_{2}(0)} \\{f_{1}(1)} \\{f_{2}(1)} \\\vdots\end{bmatrix}$

Based on the matrix [F′] a corresponding Jacobian matrix [J′] can becalculated according to Equation 53:

$J^{\prime} = \begin{bmatrix}{\frac{\partial}{{\partial\cos}\;\theta_{ec}}g_{e}} & {\frac{\partial}{{\partial\sin}\;\theta_{ec}}g_{e}} & {\frac{\partial}{{\partial\cos}\;\theta_{z}}g_{e}} & {\frac{\partial}{{\partial\sin}\;\theta_{z}}g_{e}} & {\frac{\partial}{\partial t_{xbias}}g_{e}} & {\frac{\partial}{\partial t_{ybias}}g_{e}} \\{\frac{\partial}{{\partial\cos}\;\theta_{ec}}g_{d}} & {\frac{\partial}{{\partial\sin}\;\theta_{ec}}g_{d}} & {\frac{\partial}{{\partial\cos}\;\theta_{z}}g_{d}} & {\frac{\partial}{{\partial\sin}\;\theta_{z}}g_{d}} & {\frac{\partial}{\partial t_{xbias}}g_{d}} & {\frac{\partial}{\partial t_{ybias}}g_{d}} \\{\frac{\partial}{{\partial\cos}\;\theta_{ec}}{f_{1}(0)}} & {\frac{\partial}{{\partial\sin}\;\theta_{ec}}{f_{1}(0)}} & {\frac{\partial}{{\partial\cos}\;\theta_{z}}{f_{1}(0)}} & {\frac{\partial}{{\partial\sin}\;\theta_{z}}{f_{1}(0)}} & {\frac{\partial}{\partial t_{xbias}}{f_{1}(0)}} & {\frac{\partial}{\partial t_{ybias}}{f_{1}(0)}} \\{\frac{\partial}{{\partial\cos}\;\theta_{ec}}{f_{2}(0)}} & {\frac{\partial}{{\partial\sin}\;\theta_{ec}}{f_{2}(0)}} & {\frac{\partial}{{\partial\cos}\;\theta_{z}}{f_{2}(0)}} & {\frac{\partial}{{\partial\sin}\;\theta_{z}}{f_{2}(0)}} & {\frac{\partial}{\partial t_{xbias}}{f_{2}(0)}} & {\frac{\partial}{\partial t_{ybias}}{f_{2}(0)}} \\{\frac{\partial}{{\partial\cos}\;\theta_{ec}}{f(1)}} & {\frac{\partial}{{\partial\sin}\;\theta_{ec}}{f(1)}} & {\frac{\partial}{{\partial\cos}\;\theta_{z}}{f(1)}} & {\frac{\partial}{{\partial\sin}\;\theta_{z}}{f(1)}} & {\frac{\partial}{\partial t_{xbias}}{f(1)}} & {\frac{\partial}{\partial t_{ybias}}{f(1)}} \\{\frac{\partial}{{\partial\cos}\;\theta_{ec}}{f_{2}(1)}} & {\frac{\partial}{{\partial\sin}\;\theta_{ec}}{f_{2}(1)}} & {\frac{\partial}{{\partial\cos}\;\theta_{z}}{f_{2}(1)}} & {\frac{\partial}{{\partial\sin}\;\theta_{z}}{f_{2}(1)}} & {\frac{\partial}{\partial t_{xbias}}{f_{2}(1)}} & {\frac{\partial}{\partial t_{ybias}}{f_{2}(1)}} \\\vdots & \vdots & \vdots & \vdots & \vdots & \vdots\end{bmatrix}$

Investigating just the first row of [F′] and [J′] reveals how thesematrices can be used to perform the iterative process to solve for theextrinsic camera calibration values for this configuration where both anelevation angle and deflection angle are present. For example, the firstrow of [F′] provides:g _(e)=cos²θ_(ec) sin²θ_(ec)−1and the first row of [J′] provides:[2 cos θ_(ec)2 sin θ_(ec)0 0 0 0]

The Taylor series expansion of the first row of [F′] (set equal to zero)provides:0=cos²θ_(ec)+sin²θ_(ec)−1+(2 cos θ_(ec) *p ₀)+(2 sin θ_(ec) *p ₁)which can be rearranged to provide Equation 54:(2 cos θ_(ec) *p ₀)+(2 sin θ_(ec) *p ₁)=−cos²θ_(ec)−sin²θ_(ec)+1

As a result, an equation analogous to Equation 49B for this currentconfiguration for performing extrinsic camera calibration is provided byEquation 55A:

${\left\lbrack J^{\prime} \right\rbrack\begin{bmatrix}p_{0} \\p_{1} \\p_{2} \\p_{3} \\p_{4} \\p_{5}\end{bmatrix}} = {- \left\lbrack F^{\prime} \right\rbrack}$which can be manipulated to form Equation 55B:

$\begin{bmatrix}p_{0} \\p_{1} \\p_{2} \\p_{3} \\p_{4} \\p_{5}\end{bmatrix} = {- {{\left( {\left\lbrack J^{\prime} \right\rbrack^{T}\left\lbrack J^{\prime} \right\rbrack} \right)^{- 1}\left\lbrack J^{\prime} \right\rbrack}^{T}\left\lbrack F^{\prime} \right\rbrack}}$Accordingly, Equation 55B reveals that respective observations ofmultiple pallets and a respective estimate for cos θ_(ec), sin θ_(ec),cos θ_(z), sin θ_(z), t_(xbias) and t_(ybias) can be used to provide aleast squares solution for the vector of values

$\begin{bmatrix}p_{0} \\p_{1} \\p_{2} \\p_{3} \\p_{4} \\p_{5}\end{bmatrix}.$This vector of values

$\quad\begin{bmatrix}p_{0} \\p_{1} \\p_{2} \\p_{3} \\p_{4} \\p_{5}\end{bmatrix}$can be used to iteratively update the respective estimate for cosθ_(ec), sin θ_(ec), cos θ_(z), sin θ_(z), t_(xbias) and t_(ybias)according to Equation 56A:cos θ_(ec)(t+1)=cos θ_(ec)(t)+p ₀sin θ_(ec)(t+1)=sin θ_(ec)(t)+p ₁cos θ_(z)(t+1)=cos θ_(z)(t)+p ₂sin θ_(z)(t+1)=sin θ_(z)(t)+p ₃t _(xbias)(t+1)=t _(xbias)(t)+p ₄t _(ybias)(t+1)=t _(ybias)(t)+p ₅The updated estimates for cos θ_(ec), sin θ_(ec), cos θ_(z), sin θ_(z),t_(xbias) and t_(ybias) can then be used in Equation 55B to find newvalues for

$\begin{bmatrix}p_{0} \\p_{1} \\p_{2} \\p_{3} \\p_{4} \\p_{5}\end{bmatrix}.$This iterative process may be repeated until each of the calculatedvalues for

$\quad\begin{bmatrix}p_{0} \\p_{1} \\p_{2} \\p_{3} \\p_{4} \\p_{5}\end{bmatrix}$are smaller than a predetermined threshold. For example, thispredetermined threshold may be 10⁻⁶. Once the iterative processcompletes, a respective value for cos θ_(ec), sin θ_(ec), cos θ_(z), sinθ_(z), t_(xbias) and t_(ybias) has been calculated and the camera'scontribution to the elevation angle, θ_(ec), can be determined from

${\theta_{ec} = {\tan^{- 1}\left( \frac{\sin\;\theta_{ec}}{\cos\;\theta_{ec}} \right)}},$while the deflection angle, θ_(z), can be determined from

$\theta_{z} = {{\tan^{- 1}\left( \frac{\sin\;\theta_{z}}{\cos\;\theta_{z}} \right)}.}$The camera's contribution to the elevation angle, θ_(z), the deflectionangle, θ_(z), t_(xbias), and t_(ybias) comprise the extrinsic cameracalibration values for this embodiment.

A number of the above equations are nonlinear and a variety oftechniques may be employed to the iterative process to aid convergenceto a solution for the extrinsic camera calibration values. First, anallowable maximum value maxSizeofp for the norm ∥P∥ of the vector

$\lbrack P\rbrack = \begin{bmatrix}p_{0} \\p_{1} \\p_{2} \\p_{3} \\p_{4} \\p_{5}\end{bmatrix}$may be arbitrarily selected so as to limit, or adjust, the size of [P]according to the following Equation 57:

$\left\lbrack P^{\prime} \right\rbrack = \left\{ \begin{matrix}{{{maxSizeofp}*\frac{\lbrack P\rbrack}{P}},} & {{{if}\mspace{14mu}{P}} > {maxSizeofp}} \\{\lbrack P\rbrack,} & {otherwise}\end{matrix} \right.$where, for example, maxSizeofp=10;

-   -   [P′] potentially comprises a scaled vector [P]; and    -   the norm ∥P∥ of the vector [P] may be, for example, the        Euclidean norm such that ∥P∥=√{square root over (p₀ ²+p₁ ²+p₃ ²        . . . +p_(n) ²)}.

Another technique for avoiding divergence in the iterative processdescribed above involves specifying a step size that reduces the effectof the offset, or step, values (e.g., p_(n)) calculated for eachiteration. The vector

${\lbrack P\rbrack = \begin{bmatrix}p_{0} \\p_{1} \\p_{2} \\p_{3} \\p_{4} \\p_{5}\end{bmatrix}},$in this example, has six elements and, therefore, represents aparticular direction in 6-dimensional space. For convenience, thisdirection can be referred to as the Newton direction. This vector [P]also represents a “step” used to calculate the updated values forEquation 56A and can conveniently be referred to as the Newton step. Insome instances, using the full Newton step in the Newton direction tocalculate the next iterate values in the Newton-Rhapson method may notprovide the best results. While the Newton direction can be assumed tobe a correct direction in which to make the step, in some instancesreducing an amount of that step may be better than using the full Newtonstep. Accordingly, a “step-size”, λ, may be calculated that ismultiplied with [P] (where 0<λ≦1) in order to determine a modifiedNewton step (e.g., λ[P]) that can be used to calculate the updatedvalues for Equation 56A.

In particular, this technique involves performing a line search at eachiteration of the Newton-Rhapson method to determine the step size to beused when updating the values of Equation 56A. In particular, Equation56A implies using the full Newton step or, in other words, using a stepsize at each iteration of λ(t)=1. However, as mentioned, a smaller stepsize may be used that has a value between zero and one (e.g., 0<λ(t)≦1)which provides Equation 56B:cos θ_(ec)(t+1)=cos θ_(ec)(t)+λ(t)p ₀sin θ_(ec)(t+1)=sin θ_(ec)(t)+λ(t)p ₁cos θ_(z)(t+1)=cos θ_(z)(t)+λ(t)p ₂sin θ_(z)(t+1)=sin θ_(z)(t)+λ(t)p ₃t _(xbias)(t+1)=t _(xbias)(t)+λ(t)p ₄t _(ybias)(t+1)=t _(ybias)(t)+λ(t)p ₅

As known in the art, for example in Numerical Recipes in C: The Art ofScientific Computing (2nd ed.), Press, et al., 1992, CambridgeUniversity Press, New York, N.Y., USA, a line search about a point firstuses a descent direction along which an objective function will bereduced and then computes a step size that decides how far to move inthat direction. The step size represents a distance from a startingpoint that minimizes, or generally reduces, the objective function adesired amount. One example objective function, [E], useful in theabove-described process for extrinsic camera calibration is provided byEquation 58:

$\lbrack E\rbrack = {{\frac{1}{2}\left\lbrack F^{\prime} \right\rbrack}^{T}\left\lbrack F^{\prime} \right\rbrack}$

The goal of the line search technique is to find a step size λ such that[E] has decreased a sufficient amount relative to a previous value. Asdescribed later, the “sufficiency” of the decrease amount is defined byacceptance conditions for a particular line search strategy. In general,the line search first calculates a value based on [E] using a fullNewton step (i.e., λ=1) and determines if the acceptance conditions aremet. If so, then the line search is complete. If not, then in eachsubsequent iteration the line search backtracks along the line thatextends in the Newton direction by using a smaller value for λ. Thisprocess can repeat with smaller and smaller values of λ until theacceptance conditions are satisfied.

A variety of different line search techniques may be used such as, forexample, an interpolated line search, a bisection line search, aRegula-Falsi line search, a secant line search, and others. One exampleline search technique, the Armijo Rule line search, is more fullyexplained below. In this example, λ(t) refers to the eventual step sizethat is calculated for a current iteration of the Newton-Rhapson method.However, the line search method itself is also iterative and an indexsymbol “n” refers to a current iteration of the line search.

As mentioned above, the line search first calculates a value based onthe objective function [E]. The function [E] at a particular iterationof the Newton-Rhapson method can be calculated using current iteratevalues and a current Newton step but with different test values for astep size λ. Thus, for an iteration t, the objective function [E] can beconsidered a function that is dependent on the value of a step size λ.In particular, the iterate values calculated by the Newton-Rhapsonmethod, in this particular configuration of the imaging camera 130, fromEquation 56B along with the Newton step [P] can be used to defineEquation 56C:

[V(n + 1)] = [V(n)] + λ(n)[P]${{where}\mspace{14mu}\left\lbrack {V\left( {n + 1} \right)} \right\rbrack} = {{\begin{bmatrix}{\cos\;{\theta_{ec}\left( {n + 1} \right)}} \\{\sin\;{\theta_{ec}\left( {n + 1} \right)}} \\{\cos\;{\theta_{z}\left( {n + 1} \right)}} \\{\sin\;{\theta_{z}\left( {n + 1} \right)}} \\{t_{xbias}\left( {n + 1} \right)} \\{t_{ybias}\left( {n + 1} \right)}\end{bmatrix}\mspace{14mu}{{and}\mspace{14mu}\left\lbrack {V(n)} \right\rbrack}} = \begin{bmatrix}{\cos\;{\theta_{{ec}\;}(n)}} \\{\sin\;{\theta_{ec}(n)}} \\{\cos\;{\theta_{z}(n)}} \\{\sin\;{\theta_{z}(n)}} \\{t_{xbias}(n)} \\{t_{ybias}(n)}\end{bmatrix}}$

Equation 56C reveals how the values [V(n)] that are used to determine[F] (and, thus, also determine [E]) can change based on a particularstep size λ(n) that is chosen. In particular, in Equation 56C, thevalues for [V(n)] correspond to the iterate values for the currentiteration of the Newton-Rhapson method and [P] is the Newton step fromthe current iteration of the Newton-Rhapson method. Accordingly, for aparticular iteration, n, of the line search a changing value for thestep size λ(n) is tested that results in different values for [V(n+1)]that are used when calculating a value for [E]. Thus, a line searchfunction G(λ), for a particular iteration of the Newton-Rhapson methodcan be defined according to:G(λ)=[E]

where G (λ) denotes that G is a function of a step size, λ, and [E]denotes a value of the objective function [E] for iteration t of theNewton-Rhapson method.

A first order approximation of G(λ) is given by Equation 56D:

${\hat{G}(\lambda)} = {{G(0)} + {{\lambda ɛ}\frac{\mathbb{d}{G(0)}}{\mathbb{d}\lambda}}}$

Using a full Newton step, the line search starts with n=1 and λ(1)=1.The line search iteratively determines if the acceptance condition:G(λ(n))≦Ĝ(λ(n)) is satisfied. If so, then λ(n) is considered anacceptable step size and used as λ(t) in Equation 56B. If the inequalityfails, however, then the step size is reduced according to:

${\lambda\left( {n + 1} \right)} = \frac{\lambda(n)}{2}$

and the line search continues with a next iteration. The value for ε inEquation 56D can be arbitrarily chosen to be between 0 and 1 such as,for example ε=0.5.

Thus, the previously described Newton-Rhapson iterative processdescribed above with respect to Equation 56A may be modified for eachiteration, t, to help ensure convergence by first calculating thesolution for Equation 55B, as previously described, according to:

$\begin{bmatrix}p_{0} \\p_{1} \\p_{2} \\p_{3} \\p_{4} \\p_{5}\end{bmatrix} = {- {{\left( {\left\lbrack J^{\prime} \right\rbrack^{T}\left\lbrack J^{\prime} \right\rbrack} \right)^{- 1}\left\lbrack J^{\prime} \right\rbrack}^{T}\left\lbrack F^{\prime} \right\rbrack}}$However, before using these values to update the values for cos θ_(ec),sin θ_(ec), cos θ_(z), sin θ_(z), t_(xbias) and t_(ybias) to be used inthe next iteration, (t+1), they are adjusted using the techniques justdescribed. In particular, the values for [P] can potentially be limitedaccording to Equation 57:

$\left\lbrack P^{\prime} \right\rbrack = \left\{ \begin{matrix}{{{maxSizeofp}*\frac{\lbrack P\rbrack}{P}},} & {{{if}\mspace{14mu}{P}} > {maxSizeofp}} \\{\lbrack P\rbrack,} & {otherwise}\end{matrix} \right.$to produce limited offset values

$\quad\begin{bmatrix}p_{0}^{\prime} \\p_{1}^{\prime} \\p_{2}^{\prime} \\p_{3}^{\prime} \\p_{4}^{\prime} \\p_{5}^{\prime}\end{bmatrix}$The vector [P′] is considered to specify a full Newton step and Newtondirection for a current iteration of the Newton-Rhapson method.Additionally, using the values, a line search using the function [E] forthe current iteration, t, may be performed in order to calculate a stepsize λ(t) to be used when updating the values for cos θ_(ec), sinθ_(ec), cos θ_(z), sin θ_(z), t_(xbias) and t_(ybias). Thus, ananalogous equation for Equation 56B is provided by Equation 56E:cos θ_(ec)(t+1)′=cos θ_(ec)(t)+λ(t)p ₀′sin θ_(ec)(t+1)′=sin θ_(ec)(t)+λ(t)p ₁′cos θ_(z)(t+1)′=cos θ_(z)(t)+λ(t)p ₂′sin θ_(z)(t+1)′=sin θ_(z)(t)+λ(t)p ₃′t _(xbias)(t+1)′=t _(xbias)(t)+λ(t)p ₄′t _(ybias)(t+1)′=t _(ybias)(t)+λ(t)p ₅′

It is these new values for cos θ_(ec)′, sin θ_(ec)′, cos θ_(z)′, sinθ_(z)′, t_(xbias)′ and t_(ybias)′ that are then used in the nextiteration of the process for extrinsic camera calibration.

A third configuration of calibrating the imaging camera 130 involves afull rotation matrix that includes a non-zero value for θ_(x), theelevation angle, θ_(y), the pitch angle, θ_(z) the deflection angle,t_(xbias), and t_(ybias). Similar to Equation 30F, the followingresidual equations can be defined to address the elements of a fullrotation matrix for each potential pallet object.f ₁(k)=(a _(k) X _(w) +b _(k) Y _(w) +c _(k) Z _(w) −f _(x) t _(xbias))f ₂(k)=(d _(k) X _(w) +e _(k) Y _(w) +f _(k) Z _(w) −f _(y)(t _(y) +t_(ybias)))  Equation 59A:where:a _(k) =x _(k) r ₂₀ −f _(x) r ₀₀ −x ₀ r ₂₀b _(k) =x _(k) r ₂₁ −f _(x) r ₀₁ −x ₀ r ₂₁c _(k) =x _(k) r ₂₂ −f _(x) r ₀₂ −x ₀ r ₂₂d _(k) =y _(k) r ₂₀ −f _(y) r ₁₀ −y ₀ r ₂₀e _(k) =y _(k) r ₂₁ −f _(y) r ₁₁ −y ₀ r ₂₁f _(k) =y _(k) r ₂₂ −f _(y) r ₁₂ −y ₀ r ₂₂Additional residual equations can be provided by Equation 59B:g _(e)=cos²θ_(ec)+sin²θ_(ec)−1g _(p)=cos²θ_(y) sin²θ_(y)−1g _(d)=cos²θ_(z)+sin²θ_(z)−1where θ_(x)=θ_(ec)+θ_(ef).

Using these residual equations produces analogous matrices to Equations52 and 53.

For example:

$F^{''} = \begin{bmatrix}g_{e} \\g_{p} \\g_{d} \\{f_{1}(0)} \\{f_{2}(0)} \\{f_{1}(1)} \\{f_{2}(1)} \\\vdots\end{bmatrix}$ $J^{''} = \begin{bmatrix}{\frac{\partial}{{\partial\cos}\;\theta_{ec}}g_{e}} & {\frac{\partial}{{\partial\sin}\;\theta_{ec}}g_{e}} & {\frac{\partial}{{\partial\cos}\;\theta_{y}}g_{e}} & {\frac{\partial}{{\partial\sin}\;\theta_{y}}g_{e}} & {\frac{\partial}{{\partial\cos}\;\theta_{z}}g_{e}} & {\frac{\partial}{{\partial\sin}\;\theta_{z}}g_{e}} & {\frac{\partial}{\partial t_{xbias}}g_{e}} & {\frac{\partial}{\partial t_{ybias}}g_{e}} \\{\frac{\partial}{{\partial\cos}\;\theta_{ec}}g_{p}} & {\frac{\partial}{{\partial\sin}\;\theta_{ec}}g_{p}} & {\frac{\partial}{{\partial\cos}\;\theta_{y}}g_{p}} & {\frac{\partial}{{\partial\sin}\;\theta_{y}}g_{p}} & {\frac{\partial}{{\partial\cos}\;\theta_{z}}g_{p}} & {\frac{\partial}{{\partial\sin}\;\theta_{z}}g_{p}} & {\frac{\partial}{\partial t_{xbias}}g_{p}} & {\frac{\partial}{\partial t_{ybias}}g_{p}} \\{\frac{\partial}{{\partial\cos}\;\theta_{ec}}g_{d}} & {\frac{\partial}{{\partial\sin}\;\theta_{ec}}g_{d}} & {\frac{\partial}{{\partial\cos}\;\theta_{y}}g_{d}} & {\frac{\partial}{{\partial\sin}\;\theta_{y}}g_{d}} & {\frac{\partial}{{\partial\cos}\;\theta_{z}}g_{d}} & {\frac{\partial}{{\partial\sin}\;\theta_{z}}g_{d}} & {\frac{\partial}{\partial t_{xbias}}g_{d}} & {\frac{\partial}{\partial t_{ybias}}g_{d}} \\{\frac{\partial}{{\partial\cos}\;\theta_{ec}}{f_{1}(0)}} & {\frac{\partial}{{\partial\sin}\;\theta_{ec}}{f_{1}(0)}} & {\frac{\partial}{{\partial\cos}\;\theta_{y}}{f_{1}(0)}} & {\frac{\partial}{{\partial\sin}\;\theta_{y}}{f_{1}(0)}} & {\frac{\partial}{{\partial\cos}\;\theta_{z}}{f_{1}(0)}} & {\frac{\partial}{{\partial\sin}\;\theta_{z}}{f_{1}(0)}} & {\frac{\partial}{\partial t_{xbias}}{f_{1}(0)}} & {\frac{\partial}{\partial t_{ybias}}{f_{1}(0)}} \\{\frac{\partial}{{\partial\cos}\;\theta_{ec}}{f_{2}(0)}} & {\frac{\partial}{{\partial\sin}\;\theta_{ec}}{f_{2}(0)}} & {\frac{\partial}{{\partial\cos}\;\theta_{y}}{f_{2}(0)}} & {\frac{\partial}{{\partial\sin}\;\theta_{y}}{f_{2}(0)}} & {\frac{\partial}{{\partial\cos}\;\theta_{z}}{f_{2}(0)}} & {\frac{\partial}{{\partial\sin}\;\theta_{z}}{f_{2}(0)}} & {\frac{\partial}{\partial t_{xbias}}{f_{2}(0)}} & {\frac{\partial}{\partial t_{ybias}}{f_{2}(0)}} \\{\frac{\partial}{{\partial\cos}\;\theta_{ec}}{f(1)}} & {\frac{\partial}{{\partial\sin}\;\theta_{ec}}{f(1)}} & {\frac{\partial}{{\partial\cos}\;\theta_{y}}{f(1)}} & {\frac{\partial}{{\partial\sin}\;\theta_{y}}{f(1)}} & {\frac{\partial}{{\partial\cos}\;\theta_{z}}{f(1)}} & {\frac{\partial}{{\partial\sin}\;\theta_{z}}{f(1)}} & {\frac{\partial}{\partial t_{xbias}}{f(1)}} & {\frac{\partial}{\partial t_{ybias}}{f(1)}} \\{\frac{\partial}{{\partial\cos}\;\theta_{ec}}{f_{2}(1)}} & {\frac{\partial}{{\partial\sin}\;\theta_{ec}}{f_{2}(1)}} & {\frac{\partial}{{\partial\cos}\;\theta_{y}}{f_{2}(1)}} & {\frac{\partial}{{\partial\sin}\;\theta_{y}}{f_{2}(1)}} & {\frac{\partial}{{\partial\cos}\;\theta_{z}}{f_{2}(1)}} & {\frac{\partial}{{\partial\sin}\;\theta_{z}}{f_{2}(1)}} & {\frac{\partial}{\partial t_{xbias}}{f_{2}(1)}} & {\frac{\partial}{\partial t_{ybias}}{f_{2}(1)}} \\\vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots\end{bmatrix}$As a result, an equation analogous to Equation 49B for this currentconfiguration for performing extrinsic camera calibration is provided byEquation 60A:

${\left\lbrack J^{''} \right\rbrack\begin{bmatrix}p_{0} \\p_{1} \\p_{2} \\p_{3} \\p_{4} \\\begin{matrix}p_{5} \\p_{6} \\p_{7}\end{matrix}\end{bmatrix}} = {- \left\lbrack F^{''} \right\rbrack}$which can be manipulated to form Equation 60B:

$\begin{bmatrix}p_{0} \\p_{1} \\p_{2} \\p_{3} \\p_{4} \\\begin{matrix}p_{5} \\p_{6} \\p_{7}\end{matrix}\end{bmatrix} = {- {{\left( {\left\lbrack J^{''} \right\rbrack^{T}\left\lbrack J^{''} \right\rbrack} \right)^{- 1}\left\lbrack J^{''} \right\rbrack}^{T}\left\lbrack F^{''} \right\rbrack}}$

Accordingly, Equation 60B reveals that respective observations ofmultiple pallets and a respective estimate for cos θ_(ec), sin θ_(ec),cos θ_(y), sin θ_(y), cos θ_(z), sin θ_(z), t_(xbias) and t_(ybias) canbe used to provide a least squares solution for the vector of values

$\begin{bmatrix}p_{0} \\p_{1} \\p_{2} \\p_{3} \\p_{4} \\\begin{matrix}p_{5} \\p_{6} \\p_{7}\end{matrix}\end{bmatrix}.$This vector of values

$\quad\begin{bmatrix}p_{0} \\p_{1} \\p_{2} \\p_{3} \\p_{4} \\\begin{matrix}p_{5} \\p_{6} \\p_{7}\end{matrix}\end{bmatrix}$can be used to iteratively update the respective estimate forcos θ_(ec), sin θ_(ec), cos θ_(y), sin θ_(y), cos θ_(z), sin θ_(z),t_(xbias) and t_(ybias) according to Equation 61A:cos θ_(ec)(t+1)=cos θ_(ec)(t)+p ₀sin θ_(ec)(t+1)=sin θ_(ec)(t)+p ₁cos θ_(y)(t+1)=cos θ_(y)(t)+p ₂sin θ_(y)(t+1)=sin θ_(y)(t)+p ₃cos θ_(z)(t+1)=cos θ_(z)(t)+p ₄sin θ_(z)(t+1)=sin θ_(z)(t)+p ₅t _(xbias)(t+1)=t _(xbias)(t)+p ₆t _(ybias)(t+1)=t _(ybias)(t)+p ₇The updated estimates for cos θ_(ec), sin θ_(ec), cos θ_(y), sin θ_(y),cos θ_(z), sin θ_(z), t_(xbias) and t_(ybias) can then be used inEquation 60B to find new values for

$\begin{bmatrix}p_{0} \\p_{1} \\p_{2} \\p_{3} \\p_{4} \\\begin{matrix}p_{5} \\p_{6} \\p_{7}\end{matrix}\end{bmatrix}.$This iterative process may be repeated until each of the calculatedvalues for

$\quad\begin{bmatrix}p_{0} \\p_{1} \\p_{2} \\p_{3} \\p_{4} \\\begin{matrix}p_{5} \\p_{6} \\p_{7}\end{matrix}\end{bmatrix}$are smaller than a predetermined threshold. For example, thispredetermined threshold may be 10⁻⁶. Once the iterative processcompletes, a respective value for each of cos θ_(ec), sin θ_(ec), cosθ_(y), sin θ_(y), cos θ_(z), sin θ_(z), t_(xbias) and t_(ybias) has beencalculated and the camera's contribution to the elevation angle, θ_(ec)can be determined from

${\theta_{ec} = {\tan^{- 1}\left( \frac{\sin\;\theta_{ec}}{\cos\;\theta_{ec}} \right)}},$the pitch angle θ_(y), can be determined from

${\theta_{y} = {\tan^{- 1}\left( \frac{\sin\;\theta_{y}}{\cos\;\theta_{y}} \right)}},$and the deflection angle, θ_(z), can be determined from

$\theta_{z} = {{\tan^{- 1}\left( \frac{\sin\;\theta_{z}}{\cos\;\theta_{z}} \right)}.}$Additionally, the previously described techniques for avoidingdivergence may be applied as well to this particular configuration forperforming extrinsic camera calibration.

While the foregoing disclosure discusses illustrative aspects and/orembodiments, it should be noted that various changes and modificationscould be made herein without departing from the scope of the describedaspects and/or embodiments as defined by the appended claims. Forexample, a number of two-dimensional image analysis techniques aredescribed above for analyzing an image frame in order to identify,locate, score and prune potential pallet objects and some of theirindividual features. These techniques are equally applicable to imageframes captured using a single camera system or a dual camera system.Furthermore, although elements of the described aspects and/orembodiments may be described or claimed in the singular, the plural iscontemplated unless limitation to the singular is explicitly stated.Additionally, all or a portion of any aspect and/or embodiment may beutilized with all or a portion of any other aspect and/or embodiment,unless stated otherwise.

What is claimed is:
 1. A programmable computer-implemented method forfinding possible corners of a pallet in an image comprising: acquiring agrey scale image including one or more pallets; determining, using acomputer, horizontal cross correlations between the image and a firststep-edge template to generate a set of horizontal cross correlationresults; determining, using said computer, vertical cross correlationsbetween the image and a second step-edge template to generate a set ofvertical cross correlation results; and determining, using saidcomputer, a first set of pixels, each such pixel respectivelycorresponding to a possible first corner of the one or more pallets,using a first corner template, the set of horizontal cross correlationresults and the set of vertical cross correlation results.
 2. The methodof claim 1, further comprising: comparing, using said computer, eachpixel in the first set of pixels to a first threshold value andgenerating a second set of pixels corresponding to a second set ofpossible first corners based on the comparison between the firstthreshold value and each pixel in the first set; and sliding, using saidcomputer, a local maxima-locating window over the second set of pixelsand eliminating all pixels from the second set of pixels in eachplacement of the window except for the pixel having a maximum value suchthat a third set of pixels corresponding to a third set of possiblefirst corners are defined.
 3. The method of claim 2, wherein generatingthe second set of pixels further comprises comparing each pixel in thefirst set of pixels to the first threshold value and eliminating pixelsthat are less than the first threshold value so as to generate thesecond set of pixels.
 4. The method of claim 1, further comprising:determining, using said computer, a fourth set of pixels, each suchpixel respectively corresponding to a possible second corner of the oneor more pallets using a second corner template, the set of horizontalcross correlation results and the set of vertical cross correlationresults; comparing, using said computer, each pixel in the fourth set ofpixels to a second threshold value and generating a fifth set of pixelscorresponding to a second set of possible second corners based on thecomparison of the second threshold value and each pixel in the fourthset; and sliding, using said computer, a local maxima-locating windowover the fifth set of pixels and eliminating all pixels from the fifthset of pixels in each placement of the window except for the pixelhaving a maximum value such that a sixth set of pixels corresponding toa third set of possible second corners are defined.
 5. The method ofclaim 4, wherein generating the fifth set of pixels further comprisescomparing each pixel in the fourth set to the second threshold value andeliminating pixels that are less than the second threshold value so asto generate the fifth set of pixels.
 6. The method of claim 4, whereinsaid second corner template is defined by an S×T window and saiddetermining a fourth set of pixels comprises: generating a second cornerimage having the fourth set of pixels by sliding the second cornertemplate window concurrently over the set of horizontal crosscorrelation results and the set of vertical cross correlation resultswherein; during each placement of the second corner template window asit slides concurrently over the horizontal and vertical crosscorrelation results, receiving along one predefined inner verticalcolumn of the window one or more corresponding horizontal crosscorrelation results, negating those received horizontal crosscorrelation results, receiving along one predefined outside horizontalrow of the window one or more corresponding vertical cross correlationresults, identifying the minimum value of the cross correlation resultsin the one predefined vertical column and the one predefined horizontalrow, and assigning the identified minimum value as a respective value inthe fourth set of pixels.
 7. The method of claim 1, wherein the firstcorner template is defined by an M×N window and said determining thefirst set of pixels comprises: generating a first corner image havingthe first set of pixels by sliding the first corner template windowconcurrently over the set of horizontal cross correlation results andthe set of vertical cross correlation results.
 8. The method of claim 7,wherein determining the first set of pixels comprises: during eachplacement of the first corner template window as it slides concurrentlyover the horizontal and vertical cross correlation results, receivingalong one predefined vertical column of the window one or morecorresponding horizontal cross correlation results, receiving along onepredefined horizontal row of the window one or more correspondingvertical cross correlation results, identifying the minimum value of thecross correlation results in the one predefined vertical column and theone predefined horizontal row, and assigning the identified minimumvalue as a respective value in the first set of pixels.
 9. The method ofclaim 8, wherein said one predefined vertical column comprises oneoutside vertical column and said one predefined horizontal row comprisesone outside horizontal row.
 10. The method of claim 8, wherein saidassigning the identified minimum value comprises assigning theidentified minimum value to the pixel of the first corner image which islocated at a position corresponding to an intersection of the onepredefined vertical column and the one predefined horizontal row.
 11. Asystem for finding possible corners of a pallet in an image comprising:an image acquisition component configured to acquire a grey scale imageincluding one or more pallets; a computer configured to determinehorizontal cross correlations between the image and a first step-edgetemplate to generate a set of horizontal cross correlation results; saidcomputer further configured to determine vertical cross correlationsbetween the image and a second step-edge template to generate a set ofvertical cross correlation results; and said computer further configuredto determine a first set of pixels, each such pixel respectivelycorresponding to a possible first corner of the one or more palletsusing a first corner template, the set of horizontal cross correlationresults and the set of vertical cross correlation results.
 12. Thesystem of claim 11, wherein: said computer is further configured tocompare each pixel in the first set of pixels to a first threshold valueand to generate a second set of pixels corresponding to a second set ofpossible first corners based on the comparison between the firstthreshold value and each pixel in the first set; and said computer isfurther configured to slide a local maxima-locating window over thesecond set of pixels and to eliminate all pixels from the second set ofpixels in each placement of the window except for the pixel having amaximum value such that a third set of pixels corresponding to a thirdset of possible first corners are defined.
 13. The system of claim 12,wherein to generate the second set of pixels said computer is furtherconfigured to compare each pixel in the first set of pixels to the firstthreshold value and eliminate pixels that are less than the firstthreshold value so as to generate the second set of pixels.
 14. Thesystem of claim 11, wherein said computer is further configured to:determine a fourth set of pixels, each such pixel respectivelycorresponding to a possible second corner of the one or more pallets,using a second corner template, the set of horizontal cross correlationresults and the set of vertical cross correlation results; compare eachpixel in the fourth set of pixels to a second threshold value andgenerate a fifth set of pixels corresponding to a second set of possiblesecond corners based on the comparison of the second threshold value andeach pixel in the fourth set; and slide a local maxima-locating windowover the fifth set of pixels and eliminate all pixels from the fifth setof pixels in each placement of the window except for the pixel having amaximum value such that a sixth set of pixels corresponding to a thirdset of possible second corners are defined.
 15. The system of claim 14,wherein to generate the fifth set of pixels said computer is furtherconfigured to compare each pixel in the fourth set to the secondthreshold value and eliminate pixels that are less than the secondthreshold value so as to generate the fifth set of pixels.
 16. Thesystem of claim 14, wherein said second corner template is defined by anS×T window and to determine said first set of pixels said computer isfurther configured to: generate a second corner image having the fourthset of pixels by sliding the second corner template window concurrentlyover the set of horizontal cross correlation results and the set ofvertical cross correlation results, wherein during each placement of thesecond corner template window as it slides concurrently over thehorizontal and vertical cross correlation results, said computer isconfigured to: receive along one predefined inner vertical column of thewindow one or more corresponding horizontal cross correlation results,negate those received horizontal cross correlation results, receivealong one predefined outside horizontal row of the window one or morecorresponding vertical cross correlation results, identify the minimumvalue of the cross correlation results in the one predefined verticalcolumn and the one predefined horizontal row; and assigning theidentified minimum value as a respective value in the fourth set ofpixels.
 17. The system of claim 11, wherein the first corner template isdefined by an M×N window and to determine said first set of pixels saidcomputer is further configured to: generate a first corner image havingthe first set of pixels by sliding the first corner template windowconcurrently over the set of horizontal cross correlation results andthe set of vertical cross correlation results.
 18. The system of claim17, wherein during each placement of the first corner template window asit slides concurrently over the horizontal and vertical crosscorrelation results, said computer is further configured to: receivealong one predefined vertical column of the window one or morecorresponding horizontal cross correlation results, receive along onepredefined horizontal row of the window one or more correspondingvertical cross correlation results, identify the minimum value of thecross correlation results in the one predefined vertical column and theone predefined horizontal row; and assign the identified minimum valueas a respective value in the first set of pixels.
 19. The system ofclaim 18, wherein said one predefined vertical column comprises oneoutside vertical column and said one predefined horizontal row comprisesone outside horizontal row.
 20. The system of claim 18, wherein saidcomputer is configured to assign the identified minimum value to thepixel of the first corner image which is located at a positioncorresponding to an intersection of the one vertical column and the onehorizontal row.
 21. A computer program product for finding possiblecorners of a pallet in an image, comprising: a non-transitory computerreadable storage medium having computer readable program code embodiedtherewith, the computer readable program code comprising: computerreadable program code configured to acquire a grey scale image includingone or more pallets; computer readable program code configured todetermine horizontal cross correlations between the image and a firststep-edge template to generate a set of horizontal cross correlationresults; computer readable program code configured to determine verticalcross correlations between the image and a second step-edge template togenerate a set of vertical cross correlation results; computer readableprogram code configured to determine a first set of pixels, each suchpixel respectively corresponding to a possible first corner of the oneor more pallets, using a first corner template, the set of horizontalcross correlation results and the set of vertical cross correlationresults.
 22. The computer program product of claim 21, furthercomprising: computer readable program code configured to compare eachpixel in the first set of pixels to a first threshold value andgenerating a second set of pixels corresponding to a second set ofpossible first corners based on the comparison between the firstthreshold value and each pixel in the first set; and computer readableprogram code configured to slide a local maxima-locating window over thesecond set of pixels and eliminating all pixels from the second set ofpixels in each placement of the window except for the pixel having amaximum value such that a third set of pixels corresponding to a thirdset of possible first corners are defined.
 23. The computer programproduct of claim 22, further comprising: computer readable program codeconfigured to compare each pixel in the first set of pixels to the firstthreshold value and eliminate pixels that are less than the firstthreshold value so as to generate the second set of pixels.
 24. Thecomputer program product of claim 21, further comprising: computerreadable program code configured to determine a fourth set of pixels,each such pixel respectively corresponding to a possible second cornerof the one or more pallets using a second corner template, the set ofhorizontal cross correlation results and the set of vertical crosscorrelation results; computer readable program code configured tocompare each pixel in the fourth set of pixels to a second thresholdvalue and generating a fifth set of pixels corresponding to a second setof possible second corners based on the comparison of the secondthreshold value and each pixel in the fourth set; and computer readableprogram code configured to slide a local maxima-locating window over thefifth set of pixels and eliminate all pixels from the fifth set ofpixels in each placement of the window except for the pixel having amaximum value such that a sixth set of pixels corresponding to a thirdset of possible second corners are defined.
 25. The computer programproduct of claim 24, further comprising: computer readable program codeconfigured to compare each pixel in the fourth set to the secondthreshold value and eliminate pixels that are less than the secondthreshold value so as to generate the fifth set of pixels.
 26. Thecomputer program product of claim 24, wherein said second cornertemplate is defined by an S×T window, and further comprising: computerreadable program code configured to generate a second corner imagehaving the fourth set of pixels by sliding the second corner templatewindow concurrently over the set of horizontal cross correlation resultsand the set of vertical cross correlation results; and computer readableprogram code configured to, during each placement of the second cornertemplate window as it slides concurrently over the horizontal andvertical cross correlation results, receive along one predefined innervertical column of the window one or more corresponding horizontal crosscorrelation results, negate those received horizontal cross correlationresults, receive along one predefined outside horizontal row of thewindow one or more corresponding vertical cross correlation results,identify the minimum value of the cross correlation results in the onepredefined vertical column and the one predefined horizontal row, andassign the identified minimum value as a respective value in the fourthset of pixels.
 27. The computer program product of claim 21, wherein thefirst corner template is defined by an M×N window, and comprising:computer readable program code configured to generate a first cornerimage having the first set of pixels by sliding the first cornertemplate window concurrently over the set of horizontal crosscorrelation results and the set of vertical cross correlation results.28. The computer program product of claim 27, further comprising:computer readable program code configured to, during each placement ofthe first corner template window as it slides concurrently over thehorizontal and vertical cross correlation results, receive along onepredefined vertical column of the window one or more correspondinghorizontal cross correlation results, receive along one predefinedhorizontal row of the window one or more corresponding vertical crosscorrelation results, identify the minimum value of the cross correlationresults in the one predefined vertical column and the one predefinedhorizontal row, and assign the identified minimum value as a respectivevalue in the first set of pixels.
 29. The computer program product ofclaim 28, wherein said one predefined vertical column comprises oneoutside vertical column and said one predefined horizontal row comprisesone outside horizontal row.
 30. The computer program product of claim28, further comprising: computer readable program code configured toassign the identified minimum value to the pixel of the first cornerimage which is located at a position corresponding to an intersection ofthe one predefined vertical column and the one predefined horizontalrow.